Network management system for federated multi-site logical network

ABSTRACT

Some embodiments provide a network management system for managing a logical network that spans multiple physical sites. The network management system includes a global network manager for receiving global logical network configuration data for the multiple physical sites. The network management system includes, at each of the physical sites, (i) a local network manager for receiving a logical network configuration for the physical site from the global network manager and (ii) a set of central controllers for distributing logical network configuration data to computing devices that implement the logical network at the physical site.

BACKGROUND

As more networks move to the cloud, it is more common for onecorporation or other entity to have networks spanning multiple sites.While logical networks that operate within a single site are wellestablished, there are various challenges in having logical networksspan multiple physical sites (e.g., datacenters). The sites should beself-contained, while also allowing for data to be sent from one site toanother easily. Various solutions are required to solve these issues.

BRIEF SUMMARY

Some embodiments of the invention provide a network management systemfor managing a logical network spanning multiple federated sites (e.g.,multiple datacenters). The network management system of some embodimentsincludes (i) a global network manager that manages the entire logicalnetwork spanning all of the sites, (ii) local network managers at eachsite that directly manage the logical network at their respective sites,and (iii) central controllers at each site for distributing logicalnetwork configuration data to computing devices at the site thatimplement the logical network. The global manager receives globallogical network configuration data (e.g., from a network administrator),while the local network managers receive (i) global logical networkconfiguration data for their respective sites from the global managerand (ii) local logical network configuration data (e.g., from a networkadministrator). In some embodiments, a network management application isprovided that allows the network administrator to access the globalmanager as well as some or all of the local managers via the same userinterface (UI).

In some embodiments, the global manager executes on a computing deviceat one of the sites spanned by the logical network, and each localmanager also executes on a computing device at its respective site. Insome embodiments, the global manager executes on the same computingdevice at one of the physical sites as the local manager for that site.The global manager and the local managers are in some embodimentsseparate modules of a single application, and share a database (e.g., adistributed database) stored at the site of the global manager. Someembodiments deploy each manager at a physical site as a cluster ofmachines (e.g., virtual machines (VMs)), with each machine executing ona different computing device at the same site.

Some embodiments employ a primary global manager and a secondary globalmanager, in an active-standby arrangement. The primary global manager isasynchronously synchronized with the secondary global manager as astandby for failover scenarios. The secondary global manager executes ona different computing device (or set of computing devices), located forresiliency at a different site than the primary global manager in someembodiments, and maintains an independent database. In some embodiments,both the primary and secondary global managers are deployed as clustersof machines, each executing alongside the local managers at theirrespective sites.

The global logical network configuration, in some embodiments, isreceived by the global manager as a desired configuration (e.g.,intent-based configuration) that defines elements and policies of theglobal logical network. Specifically, in some embodiments, a networkadministrator (e.g., through the network management application)provides basic configuration intent (e.g., creation of logicalforwarding elements and connections between them, definitions ofsecurity groups and policies, etc.) and the network management systemconverts this intent into the details of global logical networkconfiguration.

In some embodiments, the desired global configuration of the logicalnetwork is expressed as a hierarchical tree (also referred to as aglobal policy tree) with nodes and connections between the nodes. Someembodiments define a root node for the global logical network (alsoreferred to as a federation) and add nodes for both physical sites andlogical network entities as child nodes of the root node. In someembodiments, a network administrator adds a physical site to thefederation (e.g., by defining the physical site at the global managerand providing the global manager with the required networking andauthentication information to access the site), and the global managerautomatically discovers the physical network structure of the site. Thatis, upon receiving the necessary information, the global managercommunicates with the local manager for the site to determine groups ofcomputing devices at the site (e.g., zones of host computers andclusters of edge devices that will be configured to implement thelogical network).

For logical network entities (e.g., logical network elements and/orpolicies), when the network administrator creates a new logical networkentity, the global manager creates one or more nodes in the policy treefor the entity. In some embodiments, these logical network entities caninclude logical network elements that span one or more sites and logicalnetwork policies that apply to those elements, and the connectionsrepresent relationships between the nodes (e.g., parent-childrelationships, logical network connections, etc.). The logical networkelements include logical forwarding elements (e.g. logical routers,logical switches, etc.), as well as logical constructs (e.g., logicalports associated with the logical forwarding elements, logical groupingsof one or more sites, and groups of logical network endpoints that shareone or more attributes). Each logical network element is implemented insome embodiments by physical forwarding elements executing on computingdevices at the sites that are spanned by that logical network element.The logical network policies include forwarding policies, servicepolicies, and security policies, and are applied in some embodiments togovern the behavior of the logical forwarding elements. The policies canbe child nodes of a logical network element node, in some embodiments(e.g., static routing policy configuration for a logical router).

As part of the logical network configuration, the global manageridentifies a span for each logical network element (e.g., logicalforwarding element, security group, etc.) and each policy (e.g., eachrule) of the global logical network. The span for a particular elementor policy may be based on an administrator directly or indirectlyspecifying a span for the particular element or policy, or theparticular element or policy inhering the span from another element(e.g., based on relationships in the global policy tree). In someembodiments, the global manager uses the span for each element or policyto determine to which local managers the configuration the element orpolicy should be provided. That is, some embodiments do not provide theentire global logical network configuration to each local manager, butinstead only provide configuration for the elements and policies thatpertain to that local manager's site (e.g., a subtree of the globalpolicy tree, corresponding to the logical network elements implementedat that site).

The local manager at each site uses the relevant portion of the globaldesired configuration, received from the global manager, to manage thelogical network at the site. In some embodiments, the local manager at aparticular site (and/or a separate management plane at the site) usesthe relevant portion of the global logical network configuration togenerate and provide configuration data to the network controllers atthe particular site. In some embodiments, these network controllersidentify computing devices at the site (e.g., host computers, whichexecute physical forwarding elements) and distribute the configurationdata to the identified computing devices. In some embodiments, localcontrollers that execute on one or more of the computing devices receivethe configuration data from the network controllers and configureforwarding elements on the computing devices. The local controllers usethe configuration data to configure the physical forwarding elements toimplement the logical network elements. Each site's controller clusteralso creates mappings between logical addresses (e.g., MAC addresses oflogical network endpoints executing on the computing devices) andphysical addresses (e.g., IP addresses of tunnel endpoints at thecomputing devices), and distributes these mappings to each computingdevice to which they are relevant, as well as to other controllerclusters at other sites that require the data.

At least a subset of the computing devices that implement the logicalnetwork at each site host logical network endpoint data compute nodes(DCNs), such as virtual machines (VMs), containers, etc. In addition,some of these computing devices also execute service machines thatperform services on logical network data traffic (e.g. firewalls, loadbalancers, etc.). Some of the computing devices are designated as edgegateways, and implement certain centralized logical forwarding elements(e.g., components of logical routers) for processing certain types ofdata traffic to and from the logical network endpoint DCNs.

To enable the network administrator(s) to configure the logical networkat the global manager and/or the local managers, some embodimentsprovide a network management client application through which theadministrator can access the network managers. This single networkmanagement application provides UIs for both accessing the globalmanager and any of the local managers in order to create and/or modifythe logical network configuration. The application provides a first UIfor accessing the global manager to configure the global logical networkspanning the group of physical sites as well as additional UIs foraccessing each local manager at each of the physical sites. The UI foraccessing the local manager at a particular site allows theadministrator to (i) modify the global logical network as implemented atthe particular site and (ii) configure a local logical network at thesite (which may be completely separate from or connected to the globallogical network).

In some embodiments, the logical network components are the same for theglobal logical network and the local logical networks, and thus the UIsfor the global manager and local managers appear as a single pane ofglass with the same UI items and display areas. In addition, in someembodiments, within the UIs an item is provided to enable the user totoggle between the UIs for the different network managers.

The UIs, in some embodiments, provide sections for configuring logicalforwarding elements (which are the same constructs for both the globaland local logical networks) and for configuring network services. Insome embodiments, for each network manager, a primary UI page providesdisplay areas for each available type of logical forwarding element(e.g., logical switches, tier-0 (T0) logical routers for connecting thelogical network to external networks, and tier-1 (T1) logical routersfor connecting logical switches without requiring processing by the T0logical routers and for providing stateful services for logical networkendpoint DCNs connected to those logical switches). These display areasare selectable in order to access a separate UI page for each type oflogical forwarding element providing additional information about theconfigured logical forwarding elements of that type and enablingcreation/modification/deletion of these logical forwarding elements.Similarly, the primary UI page includes selectable display areas withinformation about different types of configured network services (e.g.,network address translation (NAT), load balancing, firewall rules,etc.). These display areas are selectable to provide additionalinformation about the respective services and allow configuration ofthese policies in some embodiments. As noted, the different UIs for thedifferent network managers provide the same display areas for the sametypes of logical forwarding elements and logical network services (whenthose services are configured in the different logical networks).

In some embodiments, the application client directly accesses the globalmanager (so long as the administrator has provided proper authenticationinformation for the global manager). To access local managers, theglobal manager acts as a proxy—that is, the application client accessesthe local managers through the global manager in some embodiments.

This network management application allows the administrator toconfigure logical network elements and policies for a single site eitherthrough the global manager or through the local manager for that site.Through the global manager, the administrator can create site-specificlogical network elements and policies that are part of the globallogical network (and can thus be expanded later to other sites). In thiscase, the logical network element configuration data will be stored aspart of the global policy tree and pushed to the site's local managerbased on spanning to that site. In addition, the logical network elementconfiguration data is backed up to the standby global manager whencreated at the global manager.

On the other hand, if created directly at the local manager as part ofthe local logical network, the logical network element configurationdata and policies will only be part of a local policy tree stored at thelocal manager. While this local manager is a cluster in someembodiments, if the site goes down (due to, e.g., a natural disaster),the local logical network configuration data is not backed up at anothersite. However, a local network administrator that only has access to thelocal manager for that site (i.e., is not granted access to the globalmanager) can use the network management application to directlyconfigure the logical network at that site.

In some cases, conflicts may occur between globally-defined logicalnetwork configuration and locally-defined logical network configuration.For instance, in the network configuration context, an IP address usedfor a local logical router might conflict with an IP address configuredfor a logical router spanning to the logical network. In the securitycontext, a local administrator could configure a first firewall rulebased on a first security group while a global administrator configuresa second firewall rule based on a second security group. If a logicalnetwork endpoint DCN belongs to both of these security groups, then thetwo firewall rules may be in conflict. Some embodiments generallyresolve security conflicts in favor of the globally-defined policy butresolve networking conflicts in favor of the locally-definedconfiguration (with the local manager reporting these overrides of theglobal configuration to the global manager for notification and/orvalidation).

As mentioned previously, the network controllers of some embodimentsoperate at each site to, among other functions, provide configurationdata from the local manager at the site to the computing devices of thesite. In some embodiments, a cluster of network controllers (alsoreferred to as the central control plane) operate at each site.

In addition to providing the configuration data from the local managersto the computing devices (i.e., host computers and edge devices) attheir particular site, the network controllers for a particular sitegenerate certain logical network state data and provide this generatedlogical network state data to (i) the computing devices at theparticular site and (ii) the network controllers at other sites.Furthermore, the network controllers at the particular site providegenerated logical network state data to the network controllers at theother sites. As described in more detail below, this logical networkstate data in some embodiments includes logical network address tophysical network address (physical location) mapping data as well assecurity group information (e.g., network addresses of logical networkendpoint DCNs belonging to security groups).

The network controllers for a group of sites spanned by a logicalnetwork connect in a full mesh in some embodiments. In some embodiments,a site manager for the controller cluster at each site exchangescertificates and any other required authentication information with theother sites (e.g., with the site managers of the other sites). This sitemanager then provides the network controllers at its site with theinformation (e.g., IP address, certificate, etc.) so that each networkcontroller at the site has connectivity with each network controller ateach of the other sites.

In some embodiments, one controller from the cluster at each site isdesignated for sending logical network state data to each other site,and one controller from the cluster at each site is designated forreceiving the logical network state data from each other site. That is,if there are three sites, the first site separately designates (i) acontroller for sending data to the second site, (ii) a controller forsending data to the third site, (iii) a controller for receiving datafrom the second site, and (iv) a controller for receiving data from thethird site. Each of these separately designated controllers may be adifferent controller in the cluster, or there may be overlap. Forinstance, different controllers could be designated for sending statedata to different sites, and for the same remote site, differentcontrollers could be designated for sending state data to the remotesite and for receiving state data from the remote site. To make theselection, some embodiments use a slot-based sharding mechanism (e.g.,by computing a hash value modulo the number of available controllers inthe cluster). As an alternative or in addition to sharding based onsites, some embodiments shard the controller cluster based on logicalnetwork state (e.g., using one controller for sending security groupdata to a particular remote site and another controller for sendinglogical network to physical network mapping data to the particularremote site).

The logical network to physical network mappings, in some embodiments,comprises mappings of logical network layer 2 (e.g., MAC) addresses tophysical network tunnel endpoint layer 3 (e.g., IP) addresses at whichthose logical network addresses can be reached. In some embodiments,when a logical network endpoint DCN is created on a host computer, thathost computer reports the new DCN along with data about the DCN to oneof the network controllers of the cluster. This data includes the MACaddress of the DCN, which is mapped to a virtual tunnel endpoint (VTEP)of the host computer, as well as the logical switch with which the MACaddress is associated. The controller cluster provides this logical MACaddress to VTEP IP address mapping (in the context of the logicalswitch) to any other host computers in the same site that implement thelogical switch, so that physical forwarding elements on these other hostcomputers can transmit logical network data messages through the site'sphysical network to the logical network DCN.

The controller cluster for a particular site also provides the list oflogical MAC addresses associated with a logical switch to each othersite spanned by the logical switch. Logical network data messages withina site are sent via a tunnel between the VTEP on the host computer forthe source logical network endpoint DCN and the VTEP on the hostcomputer for the destination logical network endpoint DCN. To send adata message associated with a logical switch from a source hostcomputer at a first site to a destination host computer at a secondsite, the source host computer tunnels the data message to a first edgedevice implementing a logical network gateway for the logical switch inthe first site, which tunnels the data message to a second edge deviceimplementing a logical network gateway for the logical switch in thesecond site, which in turn tunnels the data message to the destinationhost computer in the second site. As such, the controllers in the secondsite do not provide the logical MAC address to VTEP mappings for thelogical switch to the controllers in the first site, but instead provide(i) the list of logical MAC addresses associated with the logical switchand located at the second site and (ii) remote tunnel endpoint (RTEP) IPaddresses for reaching the logical network gateways at the second site.The controllers at the first site provide this logical network statedata to the edge devices implementing the logical network gateways forthe logical switch at the first site. In addition, to the host computersimplementing the logical switch at the first site, the controllersprovide the list of MAC addresses located at any of the sites (otherthan the first site), along with VTEP IP addresses at which edge devicesimplementing the logical network gateways for the logical switch at thefirst site can be reached.

In addition to the logical network to physical network mapping data, thenetwork controllers of some embodiments generate and share between siteslists of logical network endpoint DCNs that belong to dynamic securitygroups. From the local manager, the controller cluster receivesdefinitions of dynamic security groups (i.e., sets of criteria forbelonging to the group). When a logical network endpoint DCN matches theset of criteria for a particular security group, the controller adds thelogical network addresses (e.g., MAC and IP addresses) of the DCN to thesecurity group. In some embodiments, the controllers use informationreceived from a host computer when the DCN is created on the hostcomputer to (i) identify to which groups the DCN belongs and (ii)identify the MAC and IP addresses to add to the lists for the identifiedgroups. For each group spanning multiple sites, the controller clustersat those sites share the list of logical network addresses belonging tothe group with each other. The controllers then provide the full list ofaddresses for each group to the host computers and/or edge devices thatenforce policy rules using the security groups.

When providing updates to the logical network state data, in someembodiments the controllers do not re-send entire lists of MAC addressesfor a given logical switch or entire lists of addresses for a particularsecurity group between sites. Instead, some embodiments send each changeto the current state as an atomic update specifying the change, therebyminimizing the amount of data that needs to be transferred betweensites. The controllers at a particular site maintain a snapshot of thecurrent logical network state (e.g., in a distributed database at thesite), and whenever this state changes (e.g., due to creation ordeletion of a DCN from a host computer in the site), each controllerthat handles sending that state to another site identifies the changeand sends the change as an update to the other site. Because thesechanges can be derived by any of the controllers in the cluster at thesending site, this site does not persist the queue of updates in someembodiments. In some embodiments, the synchronization protocol islossless, so once an update is sent from a first site to a second siteit can be assumed that the second site will process that update (andwill do so in the order that the updates are received). In addition, thecontrollers at the second site persist these updates in a distributeddatabase.

If the connection from the first (sending) site to the second (receivingsite) goes down, upon reconnection some embodiments compare (i) asnapshot of the persisted data at the second site and (ii) a snapshot ofthe state to be sent at the first site to identify the differences. Thefirst site can thus only send these differences. In some embodiments,the receiving controller computes a cryptographic hash tree based on itssnapshot and sends this hash tree to the sending controller. The sendingcontroller computes a similar hash tree based on its snapshot andcompares the two in order to identify the differences in the state. Thesending controller then sends the updates to the receiving controller,in order to bring that site up to date.

In certain cases, such as when a logical network endpoint DCN moves fromone site to another, conflicts may arise at a first site based on datareceived from two other sites. When a DCN is migrated from a second siteto a third site, or crashes in the second site and is brought back up inthe third site, in an ideal scenario (i) the controller at the secondsite is notified of the deletion of the DCN, updates its logical networkstate data accordingly, and shares this data with the first site, and(ii) the controller at the third site is notified of the DCN creation,updates its logical network state data accordingly, and shares this datawith the first site. However, if there is a connection problem at thesecond site (e.g., between the host computer and the controller clusteror between the first and second sites), then the first site will notreceive information about the deletion, and will thus end up withconflicting information (e.g., IP addresses in a security group) oncethe information from the third site is received. In some embodiments,the first site resolves these sorts of conflicts based on preferring thedata from the site with a currently operational connection and, if bothconnections are operational, preferring the data with a more recenttimestamp.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawing, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 conceptually illustrates a network management system of someembodiments.

FIGS. 2-4 conceptually illustrate sets of network managers of a networkmanagement system of some embodiments for a logical network that spansthree physical sites.

FIG. 5 conceptually illustrates two user clients that access the globalmanager and/or local managers via a proxy at the global manager.

FIG. 6 conceptually illustrates in more detail different components ofthe network management system described in FIG. 2 .

FIG. 7 conceptually illustrates a manager cluster and a distributeddatabase system at a physical site spanned by the logical network.

FIG. 8 conceptually illustrates the architecture of a local manager thatreceives data through a channel from the primary global manager.

FIG. 9 conceptually illustrates a simple example of a logical network ofsome embodiments.

FIG. 10 conceptually illustrates the logical network of FIG. 9 showingthe logical routing components of the logical routers as well as thevarious logical switches that connect to these logical components andthat connect the logical components to each other.

FIG. 11 conceptually illustrates three datacenters spanned by thelogical network of FIG. 2 with the host computers and edge devices thatimplement the logical network.

FIG. 12 conceptually illustrates an example of a global policy tree ofsome embodiments for a logical network that spans multiple physicalsites.

FIG. 13 conceptually illustrates a process of some embodiments forauto-discovering the physical structure (e.g., the edge clusters and thezones of host computers) of a physical site network.

FIG. 14 conceptually illustrates a process of some embodiments forgenerating policy subtrees from a global policy tree.

FIG. 15 illustrates a global policy subtree for a physical site based onthe global policy tree and stored at the database for local manager forthe site.

FIG. 16 conceptually illustrates a local policy tree for a site that isdistinct from the global policy subtree received from the global managerin some embodiments.

FIG. 17 conceptually illustrates a process of some embodiments forhandling a CUD event received from a user client directly at the localmanager.

FIG. 18 illustrates an example of a GUI page for viewing and modifyingthe global logical network configuration at a global manager.

FIG. 19 illustrates the selection of a location selector item in the GUIpage of FIG. 18 .

FIG. 20 illustrates that the same primary UI page of FIG. 18 only showsstatistics for one site when this option has been selected through thelocation selector item.

FIG. 21 illustrates the selection of this network manager selector itemin the GUI page of FIG. 18 .

FIG. 22 illustrates a primary GUI page for a local manager.

FIG. 23 illustrates a GUI page for T0 gateways in the global logicalnetwork.

FIG. 24 illustrates a GUI page for T0 gateways at a particular site.

FIG. 25 conceptually illustrates a full mesh of network controllersacross three sites (e.g., datacenters) in some embodiments.

FIG. 26 conceptually illustrates the architecture of a networkcontroller of some embodiments.

FIG. 27 conceptually illustrates a process of some embodiments forsetting up logical network state data exchange with another site.

FIG. 28 conceptually illustrates an example showing the flow of logicalnetwork state data between designated sender and receiver masters atthree sites.

FIG. 29 conceptually illustrates the generation and transmission oflogical network to physical network mapping data both within a site andbetween sites.

FIGS. 30A-B conceptually illustrate the generation and transmission oflists of logical network addresses for dynamic security groups bothwithin a site and between sites.

FIGS. 31A-B conceptually illustrates an update to a dynamic securitygroup at a first site and the persisting of updates at a second site.

FIG. 32 conceptually illustrates a process of some embodiments foridentifying logical network state data updates required to be sent to acontroller at a remote site after reconnection with the remote site.

FIG. 33 conceptually illustrates a process of some embodiments forresolving conflicting logical network state data.

FIG. 34 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments of the invention provide a network management systemfor managing a logical network spanning multiple federated physicalsites (e.g., multiple datacenters). The network management system ofsome embodiments includes (i) a global network manager that manages theentire logical network spanning all of the sites, (ii) local networkmanagers at each site that directly manage the logical network at theirrespective sites, and (iii) central controllers at each site fordistributing logical network configuration data to computing devices atthe site that implement the logical network. The global manager receivesglobal logical network configuration data (e.g., from a networkadministrator), while the local network managers receive (i) globallogical network configuration data for their respective sites from theglobal manager and (ii) local logical network configuration data (e.g.,from a network administrator). In some embodiments, a network managementapplication is provided that allows the network administrator to accessthe global manager as well as some or all of the local managers via thesame user interface (UI).

The logical network, in some embodiments, is a conceptual networkstructure that a network administrator (or multiple networkadministrators) define through a set of network managers. Specifically,some embodiments include a global manager as well as local managers foreach site. FIG. 1 conceptually illustrates such a network managementsystem 100 of some embodiments. This network management system 100includes a global manager 105 as well as local managers 110 and 115 ateach of two datacenters 120 and 125 that are spanned by the logicalnetwork. The first datacenter 120 includes central controllers 130 aswell as host computers 135 and edge devices 140 in addition to the localmanager 110, while the second datacenter 125 includes centralcontrollers 145 as well as host computers 150 and edge devices 155 inaddition to the local manager 115.

In some embodiments, the network administrator(s) define the logicalnetwork to span a set of physical sites (in this case the twoillustrated datacenters 120 and 125) through the global manager 105. Inaddition, any logical network constructs (such as logical forwardingelements) that span multiple datacenters are defined through the globalmanager 105. This global manager, in different embodiments, may operateat one of the datacenters (e.g., on the same machine or machines as thelocal manager at that site or on different machines than the localmanager) or at a different site.

The global manager 105 provides data to the local managers at each ofthe sites spanned by the logical network (in this case, local managers110 and 115). In some embodiments, the global manager identifies, foreach logical network construct, the sites spanned by that construct, andonly provides information regarding the construct to the identifiedsites. Thus, security groups, logical routers, etc. that only span thefirst datacenter 120 will be provided to the local manager 110 and notto the local manager 115. In addition, LFEs (and other logical networkconstructs) that are exclusive to a site may be defined by a networkadministrator directly through the local manager at that site. Thelogical network configuration and the global and local network managersare described in greater detail below.

The local manager 110 or 115 at a given site (or a management planeapplication, which may be separate from the local manager) uses thelogical network configuration data received either from the globalmanager 105 or directly from a network administrator to generateconfiguration data for the host computers 135 and 150 and the edgedevices 140 and 155 (referred to collectively in the following ascomputing devices), which implement the logical network. The localmanagers provide this data to the central controllers 130 and 145, whichdetermine to which computing devices configuration data about eachlogical network construct should be provided. In some embodiments,different LFEs (and other constructs) span different computing devices,depending on which logical network endpoints operate on the hostcomputers 135 and 150 as well as to which edge devices various LFEconstructs are assigned (as described in greater detail below).

The central controllers 130 and 145, in addition to distributingconfiguration data to the computing devices, receive physical network tological network mapping data from the computing devices in someembodiments and share this information across datacenters. For instance,in some embodiments, the central controllers 130 receive tunnel endpointto logical network address mapping data from the host computers 135, andshare this information (i) with the other host computers 135 and theedge devices 140 in the first datacenter 120 and (ii) with the centralcontrollers 145 in the second site 125 (so that the central controllers145 can share this data with the host computers 150 and/or the edgedevices 155). Similarly, in some embodiments, the central controllers130 identify members of security groups in the first datacenter 120based on information from the host computers 135 and distribute thisaggregated information about the security groups to at least the hostcomputers 135 and to the central controllers in the second site 125. Thecentral controller operations are described in greater detail below.

Regarding the global and local managers, FIGS. 2-4 conceptuallyillustrate sets of network managers of a network management system ofsome embodiments for a logical network that spans three physical sites205-215. The network management system 200 shown in FIG. 2 includes (i)a global manager 220 that manages the entire logical network spanningall of the physical sites 205-215 as well as (ii) the local managers225-235 for each of the sites that manage the logical network at theirrespective sites 205-215. Each physical site also includes centralcontrollers, host computers, and edge devices (not shown) in addition tothe local manager. In some embodiments, the global manager 220 executeson a computing device 245 at one of the sites 205 spanned by the logicalnetwork, and the local managers 225-235 also execute on computingdevices 250-255 at their respective sites 210-215.

The global manager receives a global desired configuration for thelogical network via one or more user clients 240. Each of the localmanagers 225-235 may also receive in some embodiments a (site-specific)desired configuration for the logical network via the user clients 240.The desired configuration is provided to the managers 220-235 from auser client 240 in some embodiments using a representational statetransfer (REST) application programming interface (API), and isrepresented by dashed lines in FIG. 2 . The global manager 220 alsoprovides a site-specific portion of the global desired configuration toeach of the local managers 225-235, as represented by dotted lines inFIG. 2 .

In some embodiments, as illustrated in FIG. 2 , the global manager 220executes on the same computing device 245 at a given physical site 205as the local manager 225 managing that site. The global manager and thelocal manager at the same site are in some embodiments separate modulesof a single application, and share a database (e.g., a distributeddatabase) stored at the site of the global manager. In otherembodiments, as illustrated in FIG. 3 , the global manager 220 executeson a computing device 305 at a given physical site 205 separately fromany local manager 225 managing that site (though they may neverthelessshare a distributed database). In still other embodiments, asillustrated in FIG. 4 , the global manager 220 executes on a computingdevice 405 at a separate site 410 that is not spanned by the logicalnetwork, and therefore has no local managers for that logical network.

Some embodiments employ a secondary (standby) global manager 260, in anactive-standby arrangement with the primary (active) global manager 220.The primary global manager 220 is asynchronously synchronized with thesecondary global manager 260 as a standby for failover scenarios. Thisasynchronous replication is represented by a dot-dash line in FIG. 2 .For resiliency, the secondary global manager 260 is located at adifferent physical site 210 than the site 205 where the active globalmanager 220 is located, and maintains an independent database from theprimary global manager 220. This ensures that a failover scenario due toconnectivity issues to the primary global manager's physical site doesnot also automatically affect the secondary global manager's physicalsite, and exploits the multisite architecture of the federated logicalnetwork.

The secondary global manager 260 executes in some embodiments on thesame computing device 250 as a local manager 230 managing its site 210,as illustrated in FIG. 2 . In other embodiments, as illustrated in FIG.3 , the secondary global manager 260 executes on a different computingdevice 310 at its physical site 210, separately from any local manager230 managing that site. In still other embodiments, as illustrated inFIG. 4 , the secondary global manager 260 executes on a computing device415 at a separate site 420 that is not spanned by the logical network,and therefore has no local managers (but is nevertheless different thanthe location of the active global manager 220). Even though FIGS. 2-4illustrate examples where the primary global manager 220 and thesecondary global manager 260 have identical hosting arrangements, insome embodiments any combination or permutation of hosting may beemployed as required. As just one example, the primary global manager220 may be co-located with a local manager (as in FIG. 2 ), and thesecondary global manager 260 may execute at a physical site that is notspanned by the logical network (as in FIG. 4 ).

The primary global manager 220, the secondary global manager 260, andthe local managers 225-235 are in some embodiments separate modules of asingle application, and in other embodiments are separate applications.These applications in some embodiments execute as one or more processeswithin machines that execute on host computers at each physical site.Some embodiments deploy one or more of the managers 220-235 and 260 as acluster of machines at their physical site, with each machine executingon a different computing device at the same site, as described infurther detail below with reference to FIG. 7 .

It should be noted that while the above figures illustrate the userclients 240 directly connecting to the global manager 220 and the localmanagers 225-235, in other embodiments the user clients only directlyaccess the global manager 220 (so long as the application has providedproper authentication information for the global manager). To access thelocal managers in such embodiments, the global manager acts as aproxy—that is, the application client accesses the local managersthrough the global manager in some embodiments.

FIG. 5 conceptually illustrates two user clients 505 and 510 that accessthe global manager 220 and/or local managers 225-235 via a proxy 515 atthe global manager. In some embodiments, the proxy 515 is configuredwith the information needed to connect to each of the local managers225-235 (e.g., IP address, authentication information, etc.). As shown,the first user client 505 has access to the global manager 220 as wellas all three of the local managers 225-235. The second user client 510,on the other hand, only has access to the local manager 235 for thethird site 215. In this example, the second user client might belong toa local administrator for that site, and thus this administrator is notgranted privileges to access either global manager 220 or the otherlocal managers 225-230. In different embodiments, enforcement of thesedifferent access privileges for the local managers may be at the proxy515 (by validating access credentials) or at the separate localmanagers. In some embodiments of the latter case, the proxy 515 allowsconnections from any user client through to the local managers 225-235,which perform their own authentication procedures.

FIG. 6 conceptually illustrates in more detail different components ofthe network management system 200 described in FIG. 2 . The desiredconfiguration of the logical network is received by the primary globalmanager 220 from a user client 240 (e.g., over a wide area network 242such as the Internet) and stored in a database 610 at the global manager220. This desired configuration is replicated through a dedicatedasynchronous channel to the secondary global manager 260 for storage ina separate database 612. A site-specific portion of the desiredconfiguration is also provided to each of the local managers (viadedicated asynchronous channels) 225-235 for storage in their respectivedatabases (not shown).

The global manager 220 also includes a number of additional modules,including an API processor 606 for receiving the user client input via aREST API, a core global manager service 607 that writes data to thedatabase 610, a persistent work queue 611 in the database 610 tomaintain causality for incoming create/update/delete (CUD) events, a logreplication module 630 to replicate CUD events to the database 260 atthe secondary global manager 260, a broker/span service 635 to performspan calculations on the CUD events (and the logical network elementsreferenced therein), an asynchronous replication (AR) module 640 whichincludes dedicated persistent queues 625-627 for disseminating CUDevents to different local managers at different physical sites, and sitemanagers 645 for maintaining connection parameters used by the AR moduleto establish channels to the other local managers. Each of these modulesand their functionality are described in further detail below.

In some embodiments, the databases 610 and 612 are distributed databases(e.g., a shared log) implemented across a set of storage devices at themanagers' respective physical sites. In addition, in some embodiments,the global managers 220 and 260 are implemented as a cluster of machinesexecuting on separate computing devices at its respective physical site.FIG. 7 conceptually illustrates a manager cluster and a distributeddatabase system at a physical site spanned by the logical network. Themanager cluster illustrated in this example may function in differentembodiments as a primary global manager in active mode, as a secondaryglobal manager in standby mode, or as a local manager. The manager is anapplication that runs on several manager machines 705-715 executing onseparate host computers 720-730 at the physical site, which communicatewith each other over a physical network 735 at the physical site (e.g.,a datacenter network fabric). One of the manager machines 705 isassigned as the leader for the cluster in some embodiments, which sharestasks with the other manager machines in either active-active mode(e.g., using load balancing) or active-standby mode. The cluster ofmanager machines 705-715 appears as a single logical manager to othermanagers in the logical network.

In some embodiments, the manager application that runs on the machines705-715 has separate modules for global manager and local manager, whichcan be enabled or disabled as required. Alternatively or conjunctively,in some embodiments, at least one of the machines 705-715 is a dedicatedglobal manager machine, with a separate machine (not shown) for runningan application for the local manager, executing on either the same hostcomputers 720-730, or different host computers (as described above withreference to FIGS. 2-4 ).

The manager cluster stores desired configuration data in a distributeddatabase system that is managed by one or more instances 740-750 of adatabase that execute on the host computers 720-730 in some embodiments.The database executes within the local manager machine on the host insome embodiments, though they are shown as separate in the figure forclarity. The database instances 740-750 communicate with each other overthe physical network 735 at the physical site (e.g., the datacenternetwork fabric) that is used by the manager cluster. The databaseinstances 740-750 collectively appear to the manager cluster as thesingle logical database 610. In some embodiments, the instances areshards or slices of the database. In other embodiments, each instance isa node with a full copy of the data (e.g., as illustrated in the exampleof FIG. 7 ). The redundancy allows for durability in case one of thehosts 720-730 fails.

In some embodiments, the database instances 740-750 are nodes of adistributed log that is stored on the host computers 720-730. Entries inthe distributed log provide an ordered, persisted history of updates tothe state of different logical network elements and logical networkpolicies, which the manager cluster accesses via application programminginterfaces (APIs) provided by the database instances 740-750. Thedistributed log and the database APIs are described in more detail byU.S. Pat. No. 10,540,119, which is incorporated herein by reference.

Returning to the example of FIG. 6 , data describing the global desiredconfiguration is received from the user client 240 and stored in thedatabase 610 in some embodiments using a series of transactions,initiated through a series of REST API calls from the user client 240 tothe primary global manager 220. These API calls are received andprocessed by an API processor module 606 in some embodiments, which thenprovides the received data to a manager service 607 that performs thecore functions of the global manager 220. The manager service 607 storesthe received data in the database 610. In some embodiments, the data isstored in the database in tables that store configuration parameters forthe logical network elements of the logical network. In some suchembodiments, the data in the tables is expressed as a hierarchical treeof user intent, as described below with reference to FIG. 12 .Furthermore, in some embodiments, the manager service 607 also stores(e.g., duplicates) the incoming series of transactions in a work queue611, to preserve their causality and order. The work queue is apersistent queue in the database 612 in some embodiments, and forredundancy is also replicated across the database nodes.

In some embodiments, the manager service 607 is the cluster of managermachines 705-715, as described above with reference to FIG. 7 . Asillustrated in FIG. 6 , in some embodiments the secondary global manageralso has a corresponding API processor and manager service, which areactivated in a failover scenario when the primary global manager fails(e.g., becomes unreachable) and the secondary global manager becomes theactive manager.

In some embodiments, the database 610 generates one or more updatestreams from the series of transactions. When data describing a desiredconfiguration of the logical network is received (e.g., ascreate/update/delete (CUD) events received as a series of APItransactions), this data is tagged using metadata associated with eachtransaction, such as timestamp information that can be used for dataordering and database status to prevent race conditions for access. Insome embodiments, the update stream generated by the database 610 pushesnewly-written CUD events to a log replication module 630 for replicationto the secondary global manager 260.

In some embodiments, not all data stored by the global manager in thedatabase is necessarily intended for replication. Data to be replicatedin some embodiments includes policy tables, permissions, physical siteinformation, and other data that the secondary global manager wouldrequire in order to assume active status in the event of failure of theprimary global manager. Other database tables, such as those thatpertain to managing the network in active mode, are not necessary forreplication to the secondary global manager in standby mode. Inaddition, state information about the realization status of the logicalnetwork would not need to be replicated to the secondary global manager,since the realization status would be obsolete by the time a failoverscenario occurred. Some embodiments distinguish data to be replicatedfrom data that is not to be replicated by tagging the data forreplication. As noted above, in some embodiments the database 610 isshared by the primary global manager 220 with a local manager 225 (e.g.,on the same computing device 245, as illustrated in the example of FIG.2 ). In such embodiments, the local manager also writes datacorresponding to CUD events to the database, separate from the datawritten by the global manager. These local manager events are notreplicated to the database of the secondary global manager.

The global manager 220 also includes an asynchronous replication (AR)module 640 in some embodiments, which has several persistent queues625-627. Each of these persistent queues 625-627 is dedicated fordissemination of desired configuration data to one of the local managers225-235 at each physical site 205-215. In some embodiments, thesecondary global manager 260 also has an AR module (not shown) whichincludes analogous queues.

When the primary global manager 220 receives the global desiredconfiguration for the logical network, the global manager stores variousportions of the global configuration in the persistent queues 625-627,based on the relevance of the portions to the configuration of thelogical network at the queue's corresponding physical site. In someembodiments, a broker service 635 of the global manager identifies therelevant portions the global desired configuration for each physicalsite, for example based on the span of the logical network elements, asdescribed in further detail below. Span is determined in someembodiments by a span service, which in some embodiments is part of thebroker service 635 (as depicted in FIG. 6 ) and in other embodiments isa standalone service.

In some embodiments, an asynchronous replicator (AR) module 640 at theglobal manager 220 maintains a set of asynchronous channels that connectthe primary global manager 220 to the local managers 225-235. Thesechannels are depicted as dotted lines from the persistent queues 625-627to corresponding AR modules 655-665 at the local managers 225-235.

In some embodiments, the AR module maintains a dedicated asynchronouschannel that connects the primary global manager 220 to the secondaryglobal manager 260. In other embodiments, replication to the secondaryglobal manager 260 is handled by a dedicated log replication module 630.The log replication module receives the data to be replicated from thework queue 611 in the database 612 as described above, and in someembodiments replicates the data, (as depicted by a dot-dash line)directly to the database 612 of the secondary global manager 260,bypassing the AR module. In some embodiments, the secondary globalmanager 260 also has a receiver module 642 for establishing the channelwith the primary global manager, receiving the replicated data, andwriting the data to the database 612. If the secondary global manager260 becomes active (e.g., due to failover), then in some embodiments thereceiving module 642 assumes the role of the replication module.

The AR modules 640 and 650-665 maintain the channels between thephysical sites, and in some embodiments guarantee various connectionparameters (e.g., the minimum bandwidth, the maximum roundtrip time,etc.) that are required for replication of data to the secondary globalmanager and dissemination of data to the local managers. In embodimentswhere the secondary global manager and/or the local managers areimplemented as a cluster of machines, the channels also identify theleader machine for each manager cluster.

Some embodiments execute a set of site managers 645 at the primaryglobal manager 220 that provide information (e.g., connectivitystatistics, IP addresses, etc.) about the physical sites to the ARmodule 640 to use in maintaining the channels, each site managercorresponding to one of the physical sites 205-215. The site managers645 execute separately from the AR module 640 in some embodiments (asdepicted in FIG. 6 ) or run as a sub-module of the AR module in otherembodiments. The secondary global manager 260 also executes a set ofsite managers in such embodiments (not shown), though these are notactive until a failover scenario.

As noted above, the broker service 635 enqueues CUD events to thepersistent queues 625-627 of the AR module 640. In some embodiments, theAR module 640 polls its persistent queues 625-627 to determine whenthere are new CUD events. If data (e.g., corresponding to CUD events) isfound in a queue, the AR module 640 retrieves the data (i.e., dequeuesthe CUD events) and transmits the retrieved data over the channelcorresponding to that queue to the AR module of the local manager at thecorresponding site.

FIG. 8 conceptually illustrates the architecture of a local managerreceiving data through a channel from the primary global manager 220.The local manager 230 of some embodiments includes a number of modules,including a receiving AR module 665 for receiving the data from theglobal manager 220, an ingress queue 802 of the AR module for storingreceived events, a local manager service 810 that performs the corelocal manager functionality, and a database 805 (which is a distributeddatabase in some embodiments, as described above with reference to FIG.7 ). The local manager 230 also includes in some embodiments an APIprocessor 815 to receive input via REST APIs from a user client 240, anda handler 817 to send configuration data to the control plane forrealization of the desired logical network configuration. These modulesand their functions are described in more detail below.

In the example of FIG. 8 , the local manager 230 is at a differentphysical site than the primary global manager, so the data is receivedvia a wide area network 670 (e.g., the Internet, a private network,etc.). In other embodiments, the local manager is at the same physicalsite (e.g., physical site 205), and the data is received through thelocal physical network (e.g., a datacenter fabric). As another example,if the local manager is co-located on the same host machine (e.g., asillustrated in FIG. 2 ), then the data is received in some embodimentsthrough an inter-process communication method.

As described above, in some embodiments the broker service 635 retrievesnew data from the database 610 and performs a span calculation todetermine in which queues 625-627 the data should be stored. The ARmodule 640 of the global manager 220 then retrieves the new data fromeach queue and transmits the data through dedicated channels to thecorresponding physical sites. Each channel connects the sending ARmodule 640 of the primary global manager 220 to one of the receiving ARmodules 655-665 of the local managers 225-235.

Referring to the local manager 230, the AR module 665 receives data fromthe primary global manager 220 via a dedicated channel between thesending AR module 640 and the receiving AR module 665 of local manager230. In some embodiments, the receiving AR module 665 maintains apersistent ingress queue 802 to store the received data, which in somesuch embodiments is stored in active memory. The received datacorresponds in some embodiments to one or more CUD events that eachreference one or more logical network elements that span one or morephysical sites, including the physical site 210. The order and causalityof the received data is preserved by the ingress queue 802 in someembodiments.

In some embodiments, the local manager service 810 performs validationto determine whether the CUD event associated with the received data isvalid. The validation is based on whether there is any error orinconsistency in applying the CUD event to the configuration of thelogical network at the physical site. In addition, other validations arerules in some embodiments that govern whether a logical network elementcan be updated to stretch its span to a new site. For example, prior tostretching an element to a new site, the security policies that affectthe element must be stretched to the new site first. Otherwise, aloophole is created where the security policies applicable to thelogical network element are not fully applied. In some embodiments, thevalidation is performed by the local manager service 810, whichretrieves the CUD event from the ingress queue 802 and stores it in thedatabase 805.

If the CUD event is invalid, then the local manager service generates anotification for the primary global manager of the failure to validatethe CUD event. The notification in some embodiments is a notificationevent that is queued in an egress queue (not shown) of the AR module665, to be sent back to the AR module 640 at the global manager 220 viathe same asynchronous channel (e.g., the dotted line in FIG. 8 ). Inother embodiments, the notification event is sent via an out-of-bandnotification channel. Notification events in the egress queue areretrieved and sent over the channel as part of the core functionality ofthe local manager service 810 in some embodiments. Some embodiments donot drop invalid events in some embodiments, but also persist theseevents in order to maintain causality. The invalid event is stillaccepted, and an intervention (e.g., by an administrator of the network)is required to resolve the invalidity.

In some embodiments, whether the event is valid or invalid, the event isapplied to the local desired configuration of the logical network at thephysical site. The desired configuration of the logical network isexpressed as a policy tree in some embodiments, as described in furtherdetail below. In the example of FIG. 8 , the desired configuration ofthe logical network (e.g. the configuration of the logical networkelements the span of which includes the physical site 210) is stored inthe local database 805. The validated CUD event is applied to thedesired configuration. For example, if the validated CUD event is acreate event, then a logical network element defined by the event iscreated within the desired configuration stored in the database 805. Ifthe validated CUD event is an update event, then the desiredconfiguration of a logical network element referenced by the event isupdated within the desired configuration stored in the database 805. Ifthe validated CUD event is a delete event, then a logical networkelement referenced by the event is deleted within the desiredconfiguration stored in the database 805.

The local manager uses the desired configuration of the logical networkto generate and provide configuration data to the control plane of thelogical network (e.g., a central controller or cluster of controllers ateach site). In some embodiments, these controllers identify computingdevices at the site that execute physical forwarding elements anddistribute the configuration data to the identified computing devices.In some embodiments, different logical network elements span differentcomputing devices (e.g., host computers, edge devices, etc.). Eachlogical network element is implemented in some embodiments by physicalforwarding elements executing on the identified computing devices at thesites that are spanned by that logical network element. In other words,a logical network element is implemented by at least one physicalforwarding element at each site which it spans. Some embodiments havelocal controllers (also referred to as chassis controllers) that executeon one or more of the computing devices alongside the physicalforwarding elements, and which receive the configuration data from thecontroller cluster. The local controllers use the configuration data toconfigure the physical forwarding elements to implement the logicalnetwork elements at each computing device.

The local manager may also receive a CUD event directly from a userclient 240 rather than from the global manager 220. This scenario occursfor example when a local administrator of the physical site (who may ormay not be the same as the administrator of the global federated logicalnetwork as a whole) modifies the logical network's desired configurationas implemented at the local site (e.g. by specifying a series of create,update, or delete events for logical network elements whose spanincludes the local site), or modifies a local logical network at thesite (which may connect to the global logical network). These local CUDevents are stored in the database 610 in some embodiments using a seriesof transactions, initiated through a series of REST API calls from theuser client to the primary global manager 220. The user client 240 isnot at the same physical site in some embodiments, so the local CUDevent is received by the local manager 230 over a wide-area network 242.As described above, in some embodiments the global manager acts as aproxy for the user client 240 accessing the local manager 230.

These API calls are received and processed by an API processor module815 of the local manager 230 in some embodiments, which then providesthe received data to the local manager service 810 that performs thecore functions of the local manager 230. The local manager service 810,in some embodiments, determines whether the local CUD event conflictswith the desired configuration received from the global manager 220.When the local desired configuration of the logical network elementconflicts with the globally-defined desired configuration, the localmanager service 810 applies a set of priority rules to the CUD event todetermine whether the CUD event overrides the globally-defined desiredconfiguration. For example, some embodiments only allow overriding ofthe desired configuration by a local CUD event for networking-relatedconfigurations (e.g., message forwarding rules and policies). In suchcases, the local CUD event would have priority. As another example, someembodiments prevent overrides of the desired configuration by a localCUD event for security-related configurations. In such cases, theglobally-defined desired configuration would have priority.

If there is no conflict, or the local configuration has priority, thenthe local manager service 810 applies the CUD event to the local desiredconfiguration of the logical network at the physical site that islocally stored in the database 805. For example, if the CUD event is acreate event, then a logical network element defined by the event iscreated within the local desired configuration stored in the database805. If the validated CUD event is an update event, then the desiredconfiguration of a logical network element referenced by the event isupdated within the local desired configuration stored in the database805. If the validated CUD event is a delete event, then a logicalnetwork element referenced by the event is deleted within the localdesired configuration stored in the database 805.

As noted above, the local manager 230 generates and providesconfiguration data from the desired configuration of the logical networkstored in the local database 805. In the embodiment exemplified by FIG.8 , the local manager service 810 generates the configuration data fromthe stored desired configuration and provides the generated data to ahandler module 817. The handler module 817 then distributes theconfiguration data to a central controller cluster 820 of one or morecontrollers. The controller cluster identifies host computers 850 andedge devices 855 to which to distribute the configuration data.

Before describing the global (and local) policy trees of someembodiments, the logical networks described by these policy trees, aswell as their physical implementation across multiple sites, will bedescribed further. The logical network of some embodiments may includeboth logical switches (to which logical network DCNs attach) and logicalrouters. Each LFE (e.g., logical switch or logical router) isimplemented across one or more datacenters, depending on how the LFE isdefined by the network administrator. In some embodiments, the LFEs areimplemented within the datacenters by managed forwarding elements (MFEs)executing on host computers that also host DCNs of the logical network(e.g., with the MFEs executing in virtualization software of the hostcomputers) and/or on edge devices within the datacenters. The edgedevices, in some embodiments, are computing devices that may be baremetal machines executing a datapath and/or computers on which DCNsexecute to a datapath. These datapaths, in some embodiments, performvarious gateway operations (e.g., gateways for stretching logicalswitches across datacenters, gateways for executing centralized featuresof logical routers such as performing stateful services and/orconnecting to external networks).

FIG. 9 conceptually illustrates a simple example of a logical network900 of some embodiments. This logical network 900 includes a tier-0 (T0)logical router 905, a tier-1 (T1) logical router 910, and two logicalswitches 915 and 920. Though not shown, various logical networkendpoints (e.g., VMs, containers, or other DCNs) attach to logical portsof the logical switches 915 and 920. These logical network endpointsexecute on host computers in the datacenters spanned by the logicalswitches to which they attach. In this example, both the T0 logicalrouter and the T1 logical router are defined to have a span includingthree datacenters. In some embodiments, the logical switches 915 and 920inherit the span of the logical router 905 to which they connect.

As in this example, logical routers, in some embodiments, may include T0logical routers (e.g., router 905) that connect directly to externalnetworks and T1 logical routers (e.g., router 910) that segregate a setof logical switches from the rest of the logical network and may performstateful services for endpoints connected to those logical switches.These logical routers, in some embodiments, are defined by the networkmanagers to have one or more routing components, depending on how thelogical router has been configured by the network administrator.

FIG. 10 conceptually illustrates the logical network 900 showing thelogical routing components of the logical routers 905 and 910 as well asthe various logical switches that connect to these logical componentsand that connect the logical components to each other. As shown, the T1logical router 910 includes a distributed routing component (DR) 1005 aswell as a set of centralized routing components (also referred to asservice routers, or SRs) 1010-1020. T1 logical routers, in someembodiments, may have only a DR, or may have both a DR as well as SRs.For T1 logical routers, SRs allow for centralized (e.g., stateful)services to be performed on data messages sent between (i) DCNsconnected to logical switches that connect to the T1 logical router and(ii) DCNs connected to other logical switches that do not connect to thetier-1 logical router or from external network endpoints. In thisexample, data messages sent to or from DCNs connected to logicalswitches 915 and 920 will have stateful services applied by one of theSRs 1010-1020 of the T1 logical router 910 (specifically, by the primarySR 1015).

T1 logical routers may be connected to T0 logical routers in someembodiments (e.g., T1 logical router 910 connecting to T0 logical router905). These T0 logical routers, as mentioned, handle data messagesexchanged between the logical network DCNs and external networkendpoints. As shown, the T0 logical router 905 includes a DR 1025 aswell as a set of SRs 1030-1040. In some embodiments, T0 logical routersinclude an SR (or multiple SRs) operating in each datacenter spanned bythe logical router. In some or all of these datacenters, the T0 SRsconnect to external routers 1041-1043 (or to top of rack (TOR) switchesthat provide connections to external networks).

In addition to the logical switches 915 and 920 (which span all of thedatacenters spanned by the T1 DR 1005), FIG. 10 also illustrates variousautomatically-defined logical switches. Within each datacenter, the T1DR 1005 connects to its respective local T1 SR 1010-1020 via arespective transit logical switch 1045-1055. Similarly, within eachdatacenter, the T0 DR 1025 connects to its respective local T0 SR1030-1040 via a respective transit logical switch 1060-1070. Inaddition, a router link logical switch 1075 connects the primary T1 SR1015 (that performs the stateful services for the T1 logical router) tothe T0 DR 1025. In some embodiments, similar router link logicalswitches are defined for each of the other datacenters but are marked asdown.

Lastly, the network management system also defines backplane logicalswitches that connect each set of SRs. In this case, there is abackplane logical switch 1080 connecting the three T1 SRs 1010-1020 anda backplane logical switch 1085 connecting the three T0 SRs 1030-1040.These backplane logical switches, unlike the transit logical switches,are stretched across the datacenters spanned by their respective logicalrouters. When one SR for a particular logical router routes a datamessage to another SR for the same logical router, the data message issent according to the appropriate backplane logical switch.

As mentioned, the LFEs of a logical network may be implemented by MFEsexecuting on source host computers as well as by the edge devices. FIG.11 conceptually illustrates the three datacenters 1105-1115 spanned bythe logical network 900 with the host computers 1120 and edge devices1125 that implement the logical network. VMs (in this example) or otherlogical network endpoint DCNs operate on the host computers 1120, whichexecute virtualization software for hosting these VMs. Thevirtualization software, in some embodiments, includes the MFEs such asvirtual switches and/or virtual routers. In some embodiments, one MFE(e.g., a flow-based MFE) executes on each host computer 1120 toimplement multiple LFEs, while in other embodiments multiple MFEsexecute on each host computer 1120 (e.g., one or more virtual switchesand/or virtual routers). In still other embodiments, different hostcomputers execute different virtualization software with different typesof MFEs. Within this application, “MFE” is used to represent the set ofone or more MFEs that execute on a host computer to implement LFEs ofone or more logical networks.

The edge devices 1125, in some embodiments, execute datapaths (e.g.,data plane development kit (DPDK) datapaths) that implement one or moreLFEs. In some embodiments, SRs of logical routers are assigned to edgedevices and implemented by these edge devices (the SRs are centralized,and thus not distributed in the same manner as the DRs or logicalswitches). The datapaths of the edge devices 1125 may execute in theprimary operating system of a bare metal computing device and/or executewithin a VM or other DCN (that is not a logical network endpoint DCN)operating on the edge device, in different embodiments.

In some embodiments, as shown, the edge devices 1125 connect thedatacenters to each other (and to external networks). In suchembodiments, the host computers 1120 within a datacenter can send datamessages directly to each other, but send data messages to hostcomputers 1120 in other datacenters via the edge devices 1125. When asource DCN (e.g., a VM) in the first datacenter 1105 sends a datamessage to a destination DCN in the second datacenter 1110, this datamessage is first processed by the MFE executing on the same hostcomputer 1120 as the source VM, then by an edge device 1125 in the firstdatacenter 1105, then an edge device 1125 in the second datacenter 1110,and then by the MFE in the same host computer 1120 as the destinationDCN.

As mentioned, the global desired configuration of the logical network isexpressed as a hierarchical tree (also referred to as a global policytree) with nodes and connections between the nodes in some embodiments.Some embodiments define a root node for the global logical network (alsoreferred to as a federation) and add nodes for both physical sites andlogical network entities as child nodes of the root node.

For logical network entities (e.g., logical network elements and/orpolicies), when the network administrator creates a new logical networkentity, the global manager creates one or more nodes in the policy treefor the entity. In some embodiments, these logical network entities caninclude logical network elements that span one or more sites and logicalnetwork policies that apply to those elements, and the connectionsrepresent relationships between the nodes (e.g., parent-childrelationships, logical network connections, etc.). The logical networkelements include logical forwarding elements (e.g. logical routers,logical switches, etc.), as well as logical constructs (e.g., logicalports associated with the logical forwarding elements, logical groupingsof one or more sites, and groups of logical network endpoints that shareone or more attributes). Each logical network element is implemented insome embodiments by physical forwarding elements executing on computingdevices at the sites that are spanned by that logical network element.The logical network policies include forwarding policies, servicepolicies, and security policies, and are applied in some embodiments togovern the behavior of the logical forwarding elements. The policies canbe child nodes of a logical network element node, in some embodiments(e.g., static routing policy configuration for a logical router).

The primary global manager stores the global policy tree in itsdatabase, while the secondary global manager stores a replicated globalpolicy tree in its own database. In some embodiments, the nodesrepresent logical network elements that span one or more sites andlogical network policies that apply to those elements, and theconnections represent relationships between the nodes (e.g.,parent-child relationships, logical network connections, etc.).Cross-referencing between nodes is achieved by reference to a paththrough the tree's hierarchy which provides information about the spanof each node.

FIG. 12 conceptually illustrates an example of such a global policy tree1200 of some embodiments, for a logical network that spans multiplephysical sites. In some embodiments, a global root node 1202 representsthe overall federated logical network configuration. Portions of theglobal policy tree represent logical network element, including logicalforwarding elements (e.g., logical routers, logical switches, etc.). Forexample, in FIG. 12 , the global policy tree root 1202 connects a singleTier-0 logical router T0 1205, two Tier-1 logical routers T1A 1210 andT1B 1215, and two different types of network segments. These segmentsare an overlay network segment 1220 and a VLAN segment 1225. The nodefor router T0 1205 has a number of child nodes, including static routedefinitions 1230 and locale services 1235 and 1240 referencing physicalsites A and B. In this example, the router T0 1205 also spans site C,but for the sake of simplicity the corresponding locale servicesreferencing site C are not shown in the figure. The node for router T1A1210 has a child node for a logical switch 1245. The node for router T1B1215 has a child node for a locale service 1250 referencing physicalsite A.

The locale service nodes for the T0 router and the T1 routers definethese routers' span. For example, router T0 1205 spans sites A, B, andC, while router T1B 1215 spans site A. As more locale services are addedto a T0 or T1 router, the router is stretched to the correspondingsites. Unlike router T1B 1215, router T1A 1210 does not have a localeservice child node, and instead has a reference (dashed line) to routerT0 1205. Therefore, router T1A 1210 inherits the span of router T0 1205,i.e., router T1A spans sites A, B, and C. Certain child nodes alsoinherit that span automatically in some embodiments. Accordingly, thestatic route definitions 1230 under the T0 router also span sites A, B,and C. The logical switch 1245 inherits the span of its parent routerT1A 1210, which in turn derives its span from the reference to router T01205. Therefore, logical switch 1245 also spans sites A, B, and C.

Each node in the global policy tree 1200 has multiple attributes thatdefine configuration parameters, some of which are defined by the userand others of which are inherited. In some embodiments, span is not theonly attribute that is inherited by a child node from a parent node. Forexample, certain T0 or T1 routers that span more than one site have oneof the physical sites assigned as a primary site, with the other sitesbeing secondary sites. If such a logical router has multiple servicerouter (SR) components, then the SR component at the primary site takesprecedence for certain operations. This configuration is specified(e.g., by an administrator of the network) for the router and is notpart of the configuration of the locale services under the router.

The locale service nodes 1235, 1240, and 1250 have references (dashedlines) to edge clusters 1251 and 1252 at the respective sites A and B.As noted above, in this example the TO router 1205 also spans site C,but the router's locale service for that site and therefore thecorresponding reference to an edge cluster under the site C node 1265are omitted for the sake of visual clarity. The locale service nodes areassociated in some embodiments with the service routers described abovewith reference to FIG. 10 . Edge clusters are described below withreference to site nodes. The local service nodes also have various typesof child nodes in some embodiments, defining various different types ofconfiguration information available at the respective site, includinginterfaces (e.g., logical ports), L2 VPNs, BGP services, and IPSec VPNs.Even though locale services are child nodes of other elements, they donot necessarily inherit the full span of those elements. A localeservice node has the span of the single site in some embodiments (i.e.,the site of the edge cluster node referenced by the local service node),so all child nodes only inherit the span of the single site to which thelocal service node refers.

The logical switch 1245 is shown as a child node under router T1A 1210.Such logical switches, also referred to as segments, are restricted tothe parent router if they are connected as child nodes (as in FIG. 12 ).However, in some embodiments logical switches are also directlyconnected to the global root 1202. For example, overlay segment 1220 isdirectly connected to the global root 1202, and has a reference (dashedline) to router T1B 1215. This allows the overlay segment to be moved toa different router if desired, by simply changing the reference toanother logical router at the top level below global root 1202. Theoverlay router 1220 inherits the span of router T1B 1215, e.g. site A,but the overlay router could be stretched automatically if anotherlocale service were to be added to router T1B 1215.

Another type of segment in some embodiments is a VLAN-backed segment.These are defined with respect to a transport zone, which is a group ofhost devices at a single physical site. Therefore, the VLAN-backedsegment can only span that single site where the transport zone isdefined. In some embodiments, VLAN-backed segments are used as uplinksin some embodiments, to connect a logical router to an external physicalrouter outside the logical network. In other words, the VLAN is betweenthe T0 router and the external router. Since multiple T0 routers mayconnect to same external physical router, VLAN-based segments are usedin some embodiments to distinguish their traffic. Typically, connectinga logical T0 router to physical router happens at a single physicalsite, since each site has its own connection to the wide-area network(e.g., the Internet) between the sites, i.e. a unique Internet ServiceProvider (ISP). Accordingly, VLAN backed segments provide a way oflogically isolating traffic from different TO routers to the sameexternal router, even though the T0 routers may be stretched acrossmultiple sites and overlap in their span.

In the example of FIG. 12 , VLAN segment 1225 has a reference (dashedline) to an interface 1255 of the locale service 1250 under router T1B1215. The interface 1255 is limited to the span of the locale service1250, so by connecting the VLAN segment 1225 to the interface 1255, thespan of the VLAN segment is limited to only site A as required. Ifanother locale service were to be added under router T1B, then the spanof router T1B would stretch to include the new site, but the span ofVLAN segment 1225 would be unchanged since its reference is to theinterface 1255 of the local service 1250.

Interfaces in some embodiments are uplinks or service ports. Interfacesconnect to logical switches or segments, and then logical networkendpoints (such as virtual machines, data compute nodes, or other typesof workloads) are attached to those logical switches and segments. Theseendpoints also have their own services, such as DNS, TCP, etc.

In addition, the global policy tree 1200 include nodes for each physicalsite. For example, in FIG. 12 , there are nodes for site A 1260, site B1277, and site C 1265 under the global root 1202. Each site has anenforcement point child node, under which specific resources areassigned, such as edge clusters, transport zones, etc. In the example,site A's edge cluster 1251 has incoming references from locale services1235 attached to router T0 1205 and from locale services 1250 attachedto router T1B 1215. The edge cluster 1252 at site B has an incomingreference from the locale services 1240 attached to router T0 1205. Insome embodiments, edge clusters also have children corresponding tospecific edge nodes 1253, which actually implement the SRs and executeservices such as firewalls, DHCP, etc.

In some embodiments, the global manager identifies the physicalstructure of a site (and therefore the structure of the child nodes forthe site in the global policy tree 1200) via auto-discovery, when thesite is added to the federation represented by the global root node1202. FIG. 13 conceptually illustrates a process 1300 of someembodiments for auto-discovering the physical structure (e.g., the edgeclusters and the zones of host computers) of a physical site network(e.g., a datacenter network).

As shown, the process 1300 begins by receiving (at 1305) input adding anew site to a federation of sites spanned by a logical network. At thispoint, the logical network could be defined for a group of sites, withthe new site being added to this group of sites, or the administratorcould be providing the initial group of sites to be spanned prior todefining the logical network elements. In some embodiments, a networkadministrator adds a physical site to the federation by defining thephysical site at the global manager and providing the global managerwith the required networking and authentication information to accessthe site (e.g., certificates and/or login information for the localmanager).

The process 1300 adds (at 1310) a node for the new site to the globallogical network policy tree. For instance, as shown in FIG. 12 , when anadministrator initially defines Site C, the global manager would createthe node 1265 and add this as a child node of the global root node 1202in the policy tree 1200.

Next, the process 1300 determines (at 1315) connection details forcommunicating with the local manager at the new site. These connectiondetails may include, e.g., an IP address for an interface of the localmanager that communicates with the global manager, authenticationinformation, etc. As mentioned, in some embodiments the networkadministrator provides at least some of this information. In addition,based on these connection details provided by the network administrator,in some embodiments the global manager negotiates a secure connectionwith the local manager of the new site.

With the connection set up, the process 1300 communicates (at 1320) withthe local manager to auto-discover the physical network structure at thenew site. That is, the global manager communicates with the localmanager for the new site to determine groups of computing devices at thesite, such as edge clusters and transport zones of host computers. Insome embodiments, edge clusters are groups of computing devicesdesignated for performing edge gateway services (e.g., implementing SRsfor T0 and/or T1 logical routes). Transport zones, in some embodiments,are sections of host computers in a physical site, such as racks ofservers or groups of racks connected together. In some embodiments, thelocal manager already has this information about the physical networkstructure at its site. In other embodiments, the local manager queriesthe central controllers or otherwise inventories the physical networkstructure of the site.

Finally, the process 1300 adds (at 1325) nodes to the policy tree forthe auto-discovered sections of the new site's physical network. Theprocess 1300 then ends. For instance, by reference to FIG. 12 , theglobal manager would communicate with the local manager at site C andadd the transport zone node 1280. In some embodiments, the siteenforcement point node (of which the edge cluster nodes and transportzone nodes are child nodes) is added automatically by the globalmanager.

The logical network elements also include logical constructs in someembodiments, such as domains that are logical groupings of one or moresites (e.g., geographic regions), and groups of logical networkendpoints that share one or more attributes (e.g., operating system,region, etc.). Domains are defined and represented as nodes in theglobal policy tree 1200 beneath the global root 1202. The domains aredefined in some embodiments at the global manager (e.g. by anadministrator of the logical network). Unlike sites, which represent aphysical construct, domains are a logical construct, which serve as anenvelope to group different logical entities together (e.g., forsecurity purposes). For example, firewall policies or other policymicro-segmentation applied to the domain will automatically be appliedto all groups of logical endpoints defined within the domain in someembodiments.

In some embodiments, the logical network configuration (and thereforethe global policy tree) includes different types of domains. Forexample, some domains are specific to a single physical site, and arereferred to as locations. This type of domain acts as the container forall site-wide and site-specific configuration and policies. In someembodiments, a location domain is automatically created for eachphysical site in the federated logical network and cannot be modified bythe user.

Other domains are logical groups of one or more sites and are referredto as regions. Regions can be assigned to geographic regions withmultiple sites in some embodiments. For example, in the example of FIG.6 , physical site A 205 may be in Paris, physical site B 210 in London,and physical site C 215 in New York. These correspond to the site nodesA 1260, B 1277, and C 1265 in the example of FIG. 12 , respectively. Oneregion can then be defined (e.g., Europe), which includes physical sitesA and B, and a different region defined (e.g., North America) whichincludes physical site C. This is useful for example in case there aredifferent regulatory environments (e.g., the European Union's GeneralData Protection Regulation, or GDPR). Regions and locations, like alldomains, are attached to global root 1202 and are not attached to otherdomains as child nodes. Some embodiments restrict each physical site tomembership in a single location and a single region. In other words, alocation may not have more than one physical site, and a physical sitemay not be a member of two regions.

In some embodiments, domains are only created as top-level nodes beneaththe global root 1202 and cannot be children of other domains or inheritspan from other domains. Instead, the span of a domain is manuallydefined in some embodiments at the global manager (e.g., by anadministrator of the logical network) as the sites that are members ofthe domain. The span is represented in some embodiments by a domainenforcement point, which is configured to reference the site enforcementpoint for whichever sites the domain is intended to span. For example,in FIG. 12 , the domain enforcement point for domain A 1270 referencesthe site enforcement point of site A 1260 (e.g., Paris) and the siteenforcement point of site B 1277 (e.g., London). Therefore, the domain A1270 is a region (e.g., Europe) spanning sites A and B, as well aspotentially other sites (e.g., Berlin) that are not shown in FIG. 12 .In addition, the domain enforcement point for domain B 1275 referencesthe site enforcement point of site C 1265 (e.g., New York). Therefore,the domain B 1275 spans site C. In this example, domain B 1275 is aregion (e.g., North America) that may also span other physical sites(e.g., Chicago, Los Angeles, etc.) that are not shown in FIG. 12 .Alternatively, domain B is a location that is specific to site C alone.For a given domain, the group of (one or more) site enforcement pointsthat are referenced by the domain's enforcement point is also referredto as a domain deployment map in some embodiments.

In some embodiments, logical network endpoints at each site arelogically organized into security groups which can span multiple sites.Service machines as well as managed forwarding elements executing onhost computer apply logical network policies (such as network policy1273) to the data messages exchanged between security groups ofendpoints in some embodiments, based on policy rules that are defined interms of these groups. Such security groups and network policies aredefined at the global manager through the user client 240 (e.g., by anadministrator of the logical network). In some embodiments, securitygroups and network policies are represented in the global policy tree1200 as child nodes of domains, and accordingly inherit their parentdomain's span. In some embodiments, the span of a network policy isdefined not only by its parent domain, but also by sites and/or domainswhich are referenced by the policy.

For example, in FIG. 12 , domain A 1270 has a child node correspondingto security group A 1271, which accordingly inherits a span of sites Aand B (i.e., the span defined by the domain deployment map of domain A).In addition, domain B 1275 has a child node corresponding to securitygroup B 1272, which accordingly inherits a span of site C (i.e., thespan defined by the domain deployment map of domain B). Domain A 1270also has a child node corresponding to a network policy 1273. Thenetwork policy is applicable to any groups defined under the same domain(e.g., group A 1271). In order to apply a policy to a security group,the span of the security group in some embodiments must include the spanof the policy.

In some embodiments, network policies may also refer to security groupsthat are not in the same domain. For example, the network policy 1273also references security group B 1272, which is in domain B 1275, eventhough the domain deployment map for the parent domain A 1270 does notinclude domain B.

In some embodiments, the global manager parses the global policy tree toidentify the span of each node in order to generate a policy subtree foreach physical site. The global manager identifies the span of each nodein the global policy tree, then parses the global policy tree using theidentified span for each node to generate the policy subtree for eachsite. The local manager at each site (or a management plane application,which may be separate from the local manager) uses the relevant portionof the global desired configuration, received from the global manager,along with any desired configuration received directly by the localmanager itself, to manage the logical network at the site.

FIG. 14 conceptually illustrates a process 1400 of some embodiments forgenerating policy subtrees from a global policy tree. This process 1400is performed by a global manager (e.g., the policy broker of a globalmanager) in some embodiments. The process 1400 is described in part byreference to FIG. 15 , which conceptually illustrates an example of apolicy subtree for a single site based on the global policy tree 1200shown in FIG. 12 .

As shown, the process 1400 begins by receiving (at 1405) a global policytree (e.g., the global policy tree 1200) that represents the desiredconfiguration of the multi-site logical network. It should be noted thatthe process 1400 is described in terms of determining the span of nodesthrough an entire global policy tree, as would be performed if a newsite is added to a federation of sites, and therefore the global managerneeds to provide the entire global logical network to be implemented atthe new site. In some embodiments, when new configuration data isreceived creating a new element, modifying an existing element, ordeleting an element, the global manager performs a similar process forjust the new/modified/deleted nodes of the global policy tree.

The process 1400 selects (at 1410) one of the nodes of the global policytree corresponding to one of the logical network elements. In someembodiments, the nodes are selected in an iterative fashion over thehierarchy of the global tree, starting with the top-level nodes underthe global root 1202 (e.g., logical routers, logical switches andsegments, domains, etc.). Other embodiments traverse the global policytree differently.

The process determines (at 1415) the span attribute for the selectednode. In some embodiments, the span attribute is determined by a spancalculation performed by the broker service of the global manager. Insome embodiments, the span calculation is based on the relationshipsbetween the nodes in the global policy tree. For example, when therelationship between two nodes is a parent-child relationship, the childnode may inherit its span from the parent node. In other cases, however,a child node does not inherit the entire span of its parent node. Asanother example, when the relationship between two nodes is a dependenceof one node on another node, expressed as a reference from one node tothe other, the span of the dependent node will depend on the referencednode. Some logical network elements also have a pre-defined span in someembodiments, e.g. defined by an administrator of the network, which isstored in the global manager database. For these logical networkelements, the policy broker retrieves the span attributes from thedatabase.

Based on the span attribute, the process 1400 assigns (at 1420) theselected node to a policy subtree for each physical site that is spannedby the node. These policy subtrees are stored in some embodiments in thedatabase 610. In some embodiments, the policy subtrees are associatedwith the dedicated persistent queues that correspond to each of thephysical sites.

The process 1400 determines (at 1425) if any additional nodes remain inthe global policy tree. If there are additional nodes, then the processreturns to 1410, which was described above. If there are no additionalnodes, then the process continues to 1430. At 1430, the process uses thegenerated policy subtrees to generate relevant desired configuration foreach physical site. In some embodiments, the process generates therelevant configuration from the policy subtrees by identifying portionsof the global desired configuration. The process 1400 then ends.

FIG. 15 illustrates a global policy subtree 1500 for the physical siteA, based on the global policy tree 1200, and stored at the database forlocal manager for site A. Since the subtree 1500 is specific to site A,all top-level nodes below global root 1502 with span that includes siteA are preserved, while top-level nodes that are only relevant to sites Band/or C are omitted. For example, the top-level node for site B 1277,the top-level node for site C 1265, and the top-level node for domain B1275 are all omitted, as are all their respective child nodes. Inaddition, for router T0 1205, the locale services node for site B 1240is also omitted. Router T0 1205 still spans sites A and B in someembodiments, since its span attribute is associated with its definitionat the global manager 220, not the local manager 225.

Network policy 1273 is also preserved in the global policy subtree 1500.This policy is defined under domain A 1270, so in some embodiments ithas a span of site A and site B, even though this subtree is specific tosite A. In addition, as noted above with reference to FIG. 12 , networkpolicy 1273 also references group B 1272. As a result, the span of thepolicy also includes site C, even though that site is in a differentdomain. This reference to group B 1272 is also preserved in the policysubtree 1500. In some embodiments, domain nodes (e.g., the node fordomain B 1275) are pushed to the local managers at all sites, or atleast all sites at which nodes underneath those domain nodes arerequired (as is node 1272 in this case).

In some embodiments, a local manager also stores a separate policy tree,that is generated based on desired configuration received directly atthe local manager instead of from the global manager 220. This localdesired configuration is received from a network administrator to definea logical network that is confined to that site (i.e., the span of allof the logical network elements is only the single site). In someembodiments, the logical network elements that an administrator candefine for the local logical network are of the same type as the logicalnetwork elements that the administrator can define for the globallogical network. As described below, this allows the network managementapplication via which the administrator access the global and localmanagers to provide the same UI for the different network managers. Theglobal policy tree is stored in the primary global manager database, anda replica of the global policy tree is also stored in the secondaryglobal manager database. The local policy tree, meanwhile, is notreplicated to a different site in some embodiments.

FIG. 16 conceptually illustrates a local policy tree 1600 for site A.This local policy tree is distinct from the global policy subtree 1500received from the global manager in some embodiments. In this example,the local root 1602 of the local policy tree 1600 connects a site nodefor site A, two Tier-1 logical routers T1C 1605 and T1D 1610, as well asan overlay segment 1615. The overlay segment 1615 includes a referenceto the router T1D 1610, which has a locale service corresponding to siteA.

In some embodiments, logical network elements defined in the localpolicy tree 1600 may reference logical network elements defined in theglobal policy tree 1200. For example, the node for the router T1C 1605references the node for the router T0 1205 that was defined from theglobal manager 220. As a result, data messages sent to the logicalrouter T1C 1605 can be sent to the SRs for the T0 router 1205 (e.g., toreach external networks).

As noted above, in some embodiments the nodes also represent logicalnetwork policies that apply to the logical network elements. The logicalnetwork policies include forwarding policies, service policies, andsecurity policies, and are applied in some embodiments to govern thebehavior of the logical forwarding elements (e.g., by governing thebehavior of the physical forwarding elements that implement the logicalforwarding elements).

Policies are defined in some embodiments at the global manager through auser client, e.g. by an administrator of the logical network. In someembodiments, policies are one or more service rules which are enforcedat the sites on data message flows based on a set of flow attributes.The global manager in some embodiments distributes the service rules tolocal managers at the sites at which the policy is to be enforced (i.e.,the policy's span). The policies are defined in some embodiments byreference to groups of logical network endpoints that span one or moresites (e.g., security groups, which are defined in some embodiments atthe global manager). The service rules refer to these groups in someembodiments by using a group identifier that is assigned at the globalmanager when the groups are defined. The definitions of these groups aredistributed to the sites spanned by the policies.

Through the global manager, the administrator can create site-specificlogical network elements and policies that are part of the globallogical network (and can thus be expanded later to other sites). In thiscase, the logical network element configuration data will be stored aspart of the global policy tree and pushed to the site's local managerbased on spanning to that site. In addition, the logical network elementconfiguration data is backed up to the standby global manager whencreated at the global manager.

On the other hand, if created directly at the local manager as part ofthe local logical network, the logical network element configurationdata and policies will only be part of a local policy tree stored at thelocal manager. While this local manager is a cluster in someembodiments, if the site goes down (due to, e.g., a natural disaster),the local logical network configuration data is not backed up at anothersite. However, a local network administrator that only has access to thelocal manager for that site (i.e., is not granted access to the globalmanager) can use the network management application to directlyconfigure the logical network at that site.

In some cases, conflicts may occur between globally-defined logicalnetwork configuration and locally-defined logical network configuration.For instance, in the network configuration context, an IP address usedfor a local logical router might conflict with an IP address configuredfor a logical router spanning to the logical network. In the securitycontext, a local administrator could configure a first firewall rulebased on a first security group while a global administrator configuresa second firewall rule based on a second security group. If a logicalnetwork endpoint DCN belongs to both of these security groups, then thetwo firewall rules may be in conflict. Some embodiments generallyresolve security conflicts in favor of the globally-defined policy butresolve networking conflicts in favor of the locally-definedconfiguration (with the local manager reporting these overrides of theglobal configuration to the global manager for notification and/orvalidation).

FIG. 17 conceptually illustrates a process 1700 of some embodiments forhandling a CUD event received from a user client directly at the localmanager (rather than from the global manager). This scenario occurs forexample when a local admin of the physical site (who may or may not bethe same as the administrator of the global federated logical network asa whole) modifies the logical network's desired configuration asimplemented at the local site (e.g. by specifying a series of create,update, or delete events for logical network elements whose spanincludes the local site).

The process 1700 begins by receiving (at 1705) a CUD event directly froma user client. For example, as illustrated in FIG. 8 , data describingthe CUD event is received from a user client 240 and stored directly inthe database 610 in some embodiments using a series of transactions,initiated through a series of REST API calls from the user client. Theuser client may not be located at the same physical site in someembodiments, so the CUD event is received by the local manager 230 overa wide-area network 242 (e.g., the Internet). As described above, insome embodiments the API calls are received via a proxy at the globalmanager. These API calls are received and processed by an API processormodule of the local manager in some embodiments, which then provides thereceived data to the local manager service that performs the corefunctions of the local manager.

The process 1700 then determines (at 1710) whether the CUD event (orevents) is valid. The validation is based on whether there is any erroror inconsistency in applying the CUD event to the configuration of thelogical network at the physical site. In some embodiments, thevalidation is performed by the local manager service, either directlyupon receipt from the API processor or after retrieving the event fromthe database of the local manager. If the CUD event is invalid, then theprocess 1700 generates (at 1713) a notification for the failure tovalidate the CUD event. The notification in some embodiments is anotification event provided to the user client for intervention (e.g.,by an administrator of the network). The process 1700 then ends.

If the CUD event is valid, then the process 1700 determines (at 1715)whether the event is a local event. In other words, the local managerdetermines whether the CUD event only references logical networkelements defined at the local site. These elements, if defined throughthe local manager, have no span beyond the physical site, and are notknown to the global manager. If the CUD event is a local event, then theprocess 1700 continues to 1735, which is described below.

If the CUD event is not a local event (i.e. the event references alogical network element that was defined at the global manager), thenthe process 1700 determines (at 1720) whether the event overrides theglobally-defined desired configuration of the logical network element.This determination is made in some embodiments by applying a set ofpriority rules to the CUD event to determine whether the CUD event isallowed to override the globally-defined desired configuration. Forexample, some embodiments only allow overriding of the desiredconfiguration by a local CUD event for networking-related configurations(e.g., message forwarding rules and policies) or configuration profiles(timers, etc. which are affected by the local site's parameters, such aslatency). In such cases, the local CUD event would have priority.

As another example, some embodiments prevent overrides of the desiredconfiguration by a local CUD event for security-related configurations.In such cases, the globally-defined desired configuration would havepriority. In addition, in some cases the event is an emergency-relatedevent, which is only recognized by the local manager and therefore doesoverride any related global configuration. If the event does not havepriority to override the global configuration (e.g., according to thepriority rules), then the process continues to 1717, which was definedabove.

If the the CUD event does have priority to override the globally-defineddesired configuration, then the process 1700 generates (at 1725) anotification for the primary global manager of the override event. Thenotification in some embodiments is a notification event that is queuedin an egress queue (not shown) of the AR module to be sent back to theAR module at the global manager via the same asynchronous channel (e.g.,the dotted line shown in FIG. 8 ). In other embodiments, thenotification event is sent via an out-of-band notification channel.Notification events in the egress queue are retrieved and sent over thechannel as part of the core functionality of the local manager service810 in some embodiments.

Next, the process 1700 creates (at 1730) a local copy of the logicalnetwork element the configuration of which is to be overridden by theCUD event. The original logical network element from the global managerremains as a read-only object in the local manager's database in someembodiments, while the local copy (also referred to as a shadow object)is the target of the CUD event instead of the original copy.

The process 1700 then applies (at 1735) the CUD event to the localdesired configuration of the logical network at the physical site. Ifthe event is an override, then the process applies the CUD event to theshadow copy of the object instead of the original object received fromthe global manager. The desired configuration of the logical network isexpressed as a policy tree in some embodiments, which is described infurther detail above. In the example of FIG. 8 , the desiredconfiguration of the logical network (e.g. the configuration of thelogical network elements the span of which includes the physical site210) is locally stored in the database 805. The CUD event is applied tothe local desired configuration.

For example, if the CUD event is a create event, then a logical networkelement defined by the event is created within the local desiredconfiguration stored in the database. If the validated CUD event is anupdate event, then the desired configuration of a logical networkelement referenced by the event is updated within the local desiredconfiguration stored in the database. If the validated CUD event is adelete event, then a logical network element referenced by the event isdeleted within the local desired configuration stored in the database.Finally, the process 1700 generates and distributes (at 1740)configuration data to the control plane of the logical network (e.g., acentral controller or cluster of controllers at each site). The process1700 then ends.

As described above, to enable the network administrator(s) to provide adesired configuration for the logical network at the global managerand/or the local managers, some embodiments provide a network managementclient application through which the administrator can access thenetwork managers. This single network management application providesUIs for both accessing the global manager and any of the local managersin order to create and/or modify the logical network configuration. Theapplication provides a first UI for accessing the global manager toconfigure the global logical network spanning the group of physicalsites as well as additional UIs for accessing each local manager at eachof the physical sites. The UI for accessing the local manager at aparticular site allows the administrator to (i) modify the globallogical network as implemented at the particular site and (ii) configurea local logical network at the site (which may be completely separatefrom or connected to the global logical network).

In some embodiments, the logical network components are the same for theglobal logical network and the local logical networks, and thus the UIsfor the global manager and local managers appear as a single pane ofglass with the same UI items and display areas. In addition, in someembodiments, within the UIs an item is provided to enable the user totoggle between the UIs for the different network managers.

FIG. 18 illustrates an example of a GUI page 1800 for viewing andmodifying the global logical network configuration at a global manager.The GUI for the global manager (and for the local managers), in someembodiments, provides sections for configuring logical forwardingelements and for configuring network services. The GUI page 1800 is aprimary GUI page for the global manager that provides a navigation area1805 and a network overview section 1810, as well as display areas foreach available type of logical forwarding element and/or service.

The navigation area 1805 includes selectable items grouped by type oflogical network construct in some embodiments. As shown, this navigationarea 1805 includes a “connectivity” section with different types oflogical forwarding elements (e.g., logical switches (segments), T0logical routers for connecting the logical network to external networks,and T1 logical routers for connecting logical switches without requiringprocessing by the T0 logical routers and for providing stateful servicesfor logical network endpoint DCNs connected to those logical switches).In some embodiments, each of these items is selectable to cause theapplication to access configuration information at the global managerabout these different types of logical forwarding element.

In addition, the navigation area 1805 includes a “network services”section which may include information about various different types ofnetwork services. These services may be configured to be performed atthe edge devices (e.g., as part of SR processing), at the host computersas part of distributed logical forwarding element processing, by serviceDCNs on host computers at the sites, or by third-party service machinesin different networks. In this case, only NAT services are configured,but (as shown below), some embodiments may include VPN services, loadbalancing services, forwarding policies such as firewall services orpolicy-based routing, etc. The navigation area 1805 also includes an “IPmanagement” section with selectable items related to IP addressmanagement (e.g., DNS, DHCP, IP address pools, etc.) as well as ageneric “settings” section with selectable items for modifying settings(such as the illustrated networking settings).

The GUI page 1800, as mentioned, also includes a network overviewsection 1810 that provides overview statistics for the global logicalnetwork configuration. As shown, this network overview section 1810indicates the number of T0 gateways (i.e., T0 logical routers), T1gateways (i.e., T1 logical routers), and segments (i.e., logicalswitches). In some embodiments, this section 1810 also indicatesstatistics for the configured network services (e.g., the number of NATrules configured for the logical network) and IP address management(e.g., the number of configured DNS zones and DHCP servers).

The network overview section 1810 in the global manager primary UI page1800 also includes a location selector item 1815 that enables the userto select whether the application shows network overview statistics forthe entire federation of sites (i.e., the entire logical network) oronly for a single one of the sites. FIG. 19 illustrates the selection ofthis location selector item 1815, which allows the user to choose from“All Locations” or one of the individual physical sites spanned by thelogical network (London, New York, or Paris). FIG. 20 illustrates thatthe same primary UI page 1800 only shows statistics for the New Yorksite when this option has been selected through the location selectoritem 1815. Specifically, the number of T1 gateways, segments, and DHCPservers has been reduced for the New York site.

Returning to FIG. 18 , the primary GUI page 1800 for the global manageralso includes selectable display areas for each type of logicalforwarding element configured for the global logical network. In thiscase, the display areas include a T0 gateways display area 1820, a T1gateways display area 1825, and a segments display area 1830. In someembodiments, each of these respective display areas provides additionaldetails about the logical forwarding elements of the respective typeconfigured for the logical network, and may provide visualizations ofsome of these details. For example, the T0 gateways display area 1820indicates how many of the T0 gateways are currently running BGP (e.g.,to exchange routes with external networks) and provides additionalinformation about the use of BGP on the gateways. The T1 gatewaysdisplay area 1825 provides statistics regarding the number of T1 logicalrouters per T1 gateway, and the segments display area 1830 provides abreakdown between VLAN segments (e.g., for connecting T0 SRs to externalnetworks) and overlay segments (e.g., logical switches that may bestretched between sites). Though not shown in this figure, the primaryUI page 1800 may also include selectable display areas with informationabout different types of configured network services (e.g., NAT, loadbalancing, firewall rules, etc.). In some embodiments, each of theseitems is selectable to cause the application to access configurationinformation at the global manager about these different types of logicalforwarding elements and/or network services.

Lastly, the GUI page 1800 includes a network manager selector item 1835.In some embodiments, this allows the user to quickly access othernetwork managers (e.g., any of the local managers) through the samenetwork management application client and be presented with anequivalent (similar) GUI page. FIG. 21 illustrates the selection of thisnetwork manager selector item 1835. As shown, when selected, the networkmanager selector item 1835 displays a drop-down menu 2100 showing theglobal manager (which is located in London, and is currently selected)as well as each of the local managers for London, New York, and Paris(also indicating their IP addresses, and that they are located at thosesites). In some embodiments, this drop-down menu 2100 is only availableif the user is authenticated for all of the local managers (and theglobal manager). If the user is only allowed to access, e.g., one of thelocal managers, then only information retrieved from that is shown inthe GUI, and the selector item 1835 cannot be used to access othernetwork managers.

When the user selects one of the local managers from the drop-down menu,the application client accesses that local manager to retrieve desiredstate configuration pushed to the local manager and displays a primaryGUI page for the local manager. FIG. 22 illustrates a primary GUI page2200 for the London local manager. As shown, the GUI page 2200 is laidout in the same manner as the global manager GUI page 1800 (i.e., theGUIs appear through the client application as a single pane of glass).

The GUI page 2200 includes a navigation area 2205, a network overviewsection 2210, as well as display areas for each available type oflogical forwarding element and/or service. The navigation area 2205, aswith the navigation area 1805 in the global manager GUI page 1800,includes selectable items grouped by type of logical network constructin some embodiments. As shown, this navigation area 2205 includes a“connectivity” section with different types of logical forwardingelements (e.g., segments, T0 logical routers, and T1 logical routers), a“network services” section, an “IP management” section, and a “settings”section. Similar to the navigation area for the global manager, theseitems are selectable in some embodiments to navigate to GUI pagesspecific to the type of logical network construct selected, in this casefor providing information specific to the current local site (London).

The network overview section 2210 provides overview statistics for thelocal logical network configuration in the same manner that the networkoverview section 2210 in the global manager GUI page 1800 providesstatistics for the global logical network configuration. In this case,the administrator has provided input to enable many types of networkservices (VPN, NAT, load balancers, forwarding policies) at the network,but has not yet created such services, rules, policies, etc. Similarly,the global logical forwarding elements (TO gateways, T1 gateways, etc.)have not yet been stretched to the London site, so the statistics shownone of these yet either. It should also be noted that, unlike theglobal manager page, the network overview section 2210 does not includea location selector item because only information about the particularsite is available through the local manager for that particular site.

Furthermore, the GUIs for the different network managers provide thesame display areas for the same types of logical forwarding elements andlogical network services (when those services are configured in thedifferent logical networks). Thus, the GUI page 2200 includes a T0gateways display area 2215, a T1 gateways display area 2220, and asegments display area 2225. In some cases, the local manager GUI page2200 provides the same visualization and information as does the globalmanager GUI page 1800 (e.g., BGP information for the T0 logical routers,number of T1 gateways per T0 logical gateways). On the other hand, forthe segments, the GUI page 2200 provides information about segmentsconnected to T1 gateways and/or VMs (e.g., for the overlay segments)rather than a breakdown between overlay segments and VLAN segments.

Returning to FIG. 18 , if the user selects a particular type of logicalforwarding element (or logical network service) in the navigation area1805 or a selectable display area for a particular type of logicalforwarding element, the client application requests information from theglobal manager about the logical network elements of the selected typeand displays a GUI page specific to that type of logical networkelement.

FIG. 23 illustrates a GUI page 2300 for T0 gateways in the globallogical network. A shown, this GUI page 2300 includes the navigationarea 1805 as well as a T0 gateways display section 2305, which providesmore detailed information about the T0 logical routers configured forthe logical network. In some embodiments, this section 2305 lists all ofthe T0 logical routers, providing their administrator-defined name, thehigh-availability configuration mode (e.g., active-active oractive-standby), the number of T1 logical routers linked to the T0logical router, the number of segments (e.g., overlay and/or VLANsegments) linked to the T0 logical router, the status, and any alarmsthat have been raised (e.g., due to a failure to setup a T0 SR at aparticular site). The status, in some embodiments, is only provided ifthe user requests it, in which case the global manager identifies thesites spanned by the T0 router and sends requests to the local managersat those sites to determine whether the desired state has been realizedcorrectly at those sites.

In this case, there is one T0 logical router configured, and the userhas selected to view additional information about the logical router. Asshown, this additional information corresponds in some embodiments tovarious aspects of the desired state configuration shown in the policytree described above, such as the locations (indicating at which edgecluster in the various sites the T0 SR is implemented), interfaces,routing information (e.g., static routes, IP prefix lists, route maps,etc.), BGP information (e.g., the BGP settings for each SR), and routeredistribution settings (e.g., how to redistribute routes learned at oneSR to the other SRs). Each of these subsections is selectable in someembodiments to provide additional information about the T0 gateway. Inthis figure, the location item has been selected to provide additionalinformation about the SRs at each different site (London, New York, andParis). Specifically, this display indicates to which edge cluster theSR is assigned in each site, and the mode for the SR (because the HAmode is active-active, all of the SRs are primary in this case). Theother items are also expandable to provide additional information insome embodiments.

FIG. 24 illustrates a GUI page 2400 for T0 gateways at a particular oneof the sites (Paris). This GUI page may be reached either via thenavigation area in the primary page for the local manager of the site orvia the network manager selector item on a GUI page for T0 gateways at adifferent manager (e.g., the global manager or a different localmanager). Just as the primary GUI page 2200 for a local manager mirrorsthe primary GUI page 1800 for the global manager, the T0 gateways GUIpage 2400 for a local manager mirrors the T0 gateways GUI page 2300 forthe global manager. Thus, the GUI page 2400 includes a navigation area2405 as well as a T0 gateways display section 2410, which provides moredetailed information about the T0 logical routers configured for thelogical network and spanning to the particular site. These may includeTO logical routers that span multiple sites and that are configured viathe global manager, as well as T0 logical routers configured through thelocal manager of the particular site. In some embodiments, keeping inconcert with providing similar or the same GUIs for the global managerand for the local managers, the sections of the T0 gateways displaysection 2410 are similar as to those sections in the global manager GUIpage 2300. However, rather than providing information about multiplelocations, the display section 2410 only indicates the edge cluster atthe local manager to which the T0 SR is assigned.

As mentioned previously, the network controllers of some embodimentsoperate at each site to, among other functions, provide configurationdata from the local manager at the site to the computing devices of thesite. In some embodiments, a cluster of network controllers (alsoreferred to as the central control plane) operate at each site. Thenetwork controllers for a group of sites spanned by a logical networkconnect in a full mesh in some embodiments.

FIG. 25 conceptually illustrates a full mesh of network controllersacross three sites (e.g., datacenters) 2505-2515 in some embodiments. Asshown, each of the sites 2505-2515 includes a cluster of threecontrollers, each of which communicates with the other controllers inthe cluster. That is, at the first site 2505, the three controllers2516-2518 communicate with each other; at the second site 2510, thethree controllers 2521-2523 communicate with each other; and at thethird site 2515, the three controllers 2526-2528 communicate with eachother.

Each of the sites 2505-2515 also includes host computers (and edgedevices, which are not shown in the figure) that receive configurationdata from the controllers 2516-2528. In some embodiments, each computingdevice (e.g., each host computer and/or edge device) has a masternetwork controller that is responsible for providing configuration datato that computing device, as well as receiving any runtime state changesfrom the computing device (e.g., the creation and/or deletion of logicalnetwork endpoint DCNs on the computing device). For example, in thefirst site 2505, host computer 2531 has controller 2516 as its mastercontroller and host computer 2532 has controller 2517 as its master. Inthe second site 2510, both illustrated host computers 2533 and 2534 havecontroller 2522 as their master controller. In the third site 2515, hostcomputer 2535 has controller 2526 as its master controller and hostcomputer 2536 has controller 2528 as its master controller.

In addition, each controller at each of the sites communicates with eachcontroller at each of the other sites. As shown in the figure, each ofthe controllers 2516-2518 at the first site 2505 has a connection toeach of the controllers 2521-2523 at the second site 2510 and each ofthe controllers 2526-2528 at the third site 2515. Similarly, each of thecontrollers 2521-2523 at the second site 2510 has a connection to eachof the controllers 2526-2528 at the third site 2515. Each of theseconnections is a bidirectional connection in some embodiments. However,as described below, not all of the connections are used in all cases(and some connections may be unidirectional for the provision of logicalnetwork state).

FIG. 26 conceptually illustrates the architecture of a networkcontroller 2600 of some embodiments. In some embodiments, the networkcontroller is an application operating on a computing device (e.g.,executing within a VM or container, or on a bare metal operatingsystem). The network controller 2600 includes a site mastership module2605, a site sync module 2610, a span calculator 2615, a MAC:TEP recordgenerator 2620, and a dynamic group translator 2625. The networkcontroller also accesses a distributed database 2630 for storing datareceived from controllers at remote sites as well as a storage 2632(e.g., another distributed database) for storing records of data for thelocal site. In some embodiments, the network controller 2600 has aseparate distributed database (or separate database partition) for datafrom each remote site the controllers of which provide data to thenetwork controller 2600 (or other controllers in the local cluster).Similar to the distributed database described above by reference to FIG.7 , in some embodiments the distributed database 2630 and/or the localsite records storage 2632 is stored on each of the computing devices onwhich the members of the network controller cluster operate. Each of thenetwork controllers in the cluster therefore has access to the entiredistributed database 2630 and the entire set of local site records 2632.

In some embodiments, a site manager 2635 for the controller cluster ateach site exchanges certificates and any other required authenticationinformation with the other sites (e.g., with the site managers of theother sites). This site manager 2635 then provides the networkcontrollers at its site with the information (e.g., IP address,certificate, etc.) so that each network controller at the site hasconnectivity with each network controller at each of the other sites.

In some embodiments, the site mastership module 2605 receives thisinformation from the site manager 2635 whenever a new site is added tothe logical network. The site manager gathers the requiredauthentication information and provides this information to the sitemastership module 2605. In some embodiments, one controller from thecluster at each site is designated for sending logical network statedata to each other site, and one controller from the cluster at eachsite is designated for receiving the logical network state data fromeach other site. To make the selection, the site mastership module 2605of some embodiments uses a slot-based sharding mechanism (e.g., bycomputing a hash value modulo the number of available controllers in thecluster). In some embodiments, the sharding mechanism is deterministic(e.g., based on controller and/or site identifiers), and the sitemastership module 2605 at each of the controllers in the clusterperforms the same computations to determine the sender controller andreceiver controller for communication with each other site. This processis described in further detail below by reference to FIG. 27 . As analternative or in addition to sharding based on sites, some embodimentsshard the controller cluster based on logical network state (e.g., usingone controller for sending security group data to a particular remotesite and another controller for sending logical network to physicalnetwork mapping data to the particular remote site).

When the site mastership module 2605 determines that the networkcontroller 2600 is the sender controller for a particular other site,the site sync module 2610 is responsible for communicating with thatparticular other site to provide logical network state data to theparticular site via the remote site interface 2650. As described below,in some embodiments the MAC: TEP record generator 2620 and dynamic grouptranslator 2625 retrieve the local site records 2632 and generate datato be provided to the remote sites (e.g., lists of MAC addresses locatedat the local site for logical switches spanning the local site and theparticular remote site, lists of IP and MAC addresses for DCNs at thesite belonging to various security groups). These sets of data are thenprovided to the site sync module 2610 to be provided to the remote site.

Similarly, when the site mastership module 2605 determines that thenetwork controller 2600 is the receiver controller for a particularremote site, the site sync module 2610 is responsible for communicatingwith that particular remote site to receive logical network state datafrom the remote site via the remote site interface 2650. In someembodiments, the site sync module 2610 stores the logical network statedata received from the controllers at other sites in the distributeddatabase 2630 (e.g., in the specific database or database partition forthat particular remote site).

In some embodiments, the network controller 2600 receives logicalnetwork configuration data from the local manager for the site via thelocal manager interface 2640 and stores this data in the distributeddatabase 2630. The span calculator 2615 receives this networkconfiguration data from the local manager interface 2640 (or fromanother controller in the cluster for the local site, if that othercontroller received the data from the local manager and stored the datain another shared database), and determines the computing devices towhich the data should be distributed. At the global manager level, thespan of a logical network element specifies the physical sites to whichthe configuration data for the logical network element is distributed.At the controller level for a particular site, the span specifies thecomputing devices (e.g., edge devices and/or host computers) in thatparticular site that require the configuration data for the logicalnetwork element. The span calculator 2615 identifies which logicalnetwork configuration data goes to which computing devices for which thenetwork controller 2600 is the master controller, and sends this data tothese computing devices (e.g., to local controllers on the devices) viathe computing device interface 2645.

In addition to providing the configuration data from the local managersto the computing devices (i.e., host computers and edge devices) attheir particular site, the network controllers for a particular sitegenerate certain logical network state data and provide this generatedlogical network state data to (i) the computing devices at theparticular site and (ii) the network controllers at other sites. In someembodiments, the network controller 2600 receives data from thecomputing devices at its site via the computing device interface 2645,and stores this data in the local site records storage 2632. Thisinformation includes information about logical network endpoint DCNsexecuting on the particular computing devices (e.g., MAC and IPaddresses, DCN tags, etc.) as well as information about the computingdevices themselves (e.g., tunnel endpoint (TEP) IP addresses).

In some embodiments, the MAC:TEP record generator 2620 generates logicalnetwork address to physical network address (physical location) mappingdata based on the data stored in the local site records 2632 and datafrom the remote site(s) stored in the distributed database 2630. Theserecords can include records for computing devices at the local site aswell as records to be provided to the remote site. In some embodiments,the dynamic group translator 2625 generates security group information,such as network addresses of logical network endpoint DCNs belonging tosecurity groups.

The dynamic group translator 2625 receives security group definitionsfrom the local manager and endpoint DCN information from the local siterecords 2632 and combines this information to generate the lists ofnetwork addresses (e.g., MAC and IP addresses) for different securitygroups. In some embodiments, the dynamic group translator 2625 alsocombines this data with lists of network addresses for the securitygroups received from the remote site and stored in the distributeddatabase 2630. These logical network state generation and distributionoperations are described in further detail below.

The span calculator 2615 receives generated logical network state datafrom the MAC:TEP record generator 2620 and/or the dynamic grouptranslator 2625 and determines to which computing devices at the localsite this logical network state data should be provided (e.g., based onthe logical switch and/or security group to which the logical networkstate data pertains).

In order to exchange logical network state data with controllers atother sites, as mentioned, the site mastership module 2605 on a networkcontroller identifies whether that network controller is the sendingand/or receiving master for a particular other site. FIG. 27conceptually illustrates a process 2700 of some embodiments for settingup logical network state data exchange with another site. In someembodiments, the process 2700 is performed by each controller at aparticular site when another site joins the same federation of sitesspanned by a logical network as the particular site. This process 2700would also be performed by the controllers at the new site, becausewhile the logical network state data exchange is bidirectional,mastership for each direction is handled separately in some embodiments.

As shown, the process 2700 begins by receiving (at 2705) communicationinformation for controllers at the new site. In some embodiments, thesite manager at the particular site exchanges certificates and any otherrequired authentication information with the new site, and provides thenetwork controller at its site with this information (e.g., IP address,certificate, etc.) so that the network controller has connectivity witheach network controller at the new site.

The process 2700 then performs (at 2710) a sender mastershipcalculation. In some embodiments, the controller uses a slot-basedsharding mechanism such that (i) different controllers in the clusterwill be the master sender for different other sites and (ii) eachcontroller in the cluster will perform the same calculation to determinethe master. For instance, some embodiments compute a hash value based oninformation about both controller clusters (i.e., the controllers forthe particular site and the new site), such as using controlleridentifiers, controller IP addresses, etc. While this process shards theresponsibility for sending (and receiving) logical network state databased on sites, some embodiments instead shard this responsibility basedon logical network state. For example, some embodiments have differentcontrollers in a cluster responsible for sending and/or receivingtranslated network addresses for different security groups, or logicalnetwork to physical network mapping records for different logicalswitches.

Next, the process 2700 determines (at 2715) whether it (i.e., thecontroller performing the process 2700) is designated as the sender forthe new site (i.e., whether the mastership calculation has designateditself as the master for sending logical network state data to the newsite). If not, the process 2700 ends, as a different network controllerin the cluster is the master (and therefore will be performing theremaining operations of the process 2700).

If this controller is the master, the process selects (at 2720) acontroller at the new site with which to perform a mastership check. Insome embodiments, the controller performing the process 2700 selects thecontroller at the other site randomly. In other embodiments, thecontroller performing the process 2700 uses a round-robin selectionmechanism to select the controller at the other site. Still otherembodiments may use other selection mechanisms.

The process 2700 then sends (at 2725) a mastership check message to theselected controller. This mastership check message, in some embodiments,(i) specifies that the controller sending the message is designated asthe master for sending logical network state data to the new site and(ii) requests that the controller to which the message is sent determinewhether it is designated as the master for receiving logical networkstate data from the site sending the message.

This causes the controller at the new site that receives the mastershipcheck message (the recipient controller) to perform a mastershipcalculation (for receiving rather than for sending). Based on thismastership calculation, the recipient controller sends a return messageto the sending controller. As such, the process 2700 receives (at 2730)this return message from the selected (recipient) controller. In someembodiments, this return message either (i) indicates that the selectedcontroller is designated as the receive master for the new sitevis-à-vis the existing site at which the process 2700 executes or (ii)indicates that the selected controller is not designated as the receivemaster, but specifies which controller in its cluster is designated asthe receive master.

The process 2700 determines (at 2735) whether the selected controllernode is the receive master for the new site, based on this returnmessage. If the previously selected controller (that sent the returnmessage) is not the designated receive master, then the process selects(at 2740) the controller node specified in the return message as thereceive master for the mastership check, and returns to 2725 to send themastership check message with this newly selected controller node.

This newly selected controller at the new site will receive themastership check message, perform the same deterministic calculation todetermine receive mastership, and presumably identify itself as thereceive master. The new recipient controller sends a return messageindicating that it is the receive master to the sending controllerperforming the process 2700. Once the receive master is identified(whether on the first or subsequent attempt), the process 2700 begins(at 2745) syncing logical network state data by identifying the statedata for the new site and transmitting this data to the identifiedreceive master controller. The process 2700 then ends.

The result of this process (and the corresponding receive mastershipcheck) being performed at each site is that one controller from thecluster at each site is designated for sending logical network statedata to each other site, and one controller from the cluster at eachsite is designated for receiving the logical network state data fromeach other site. That is, if there are three sites, the first siteseparately designates (i) a controller for sending data to the secondsite, (ii) a controller for sending data to the third site, (iii) acontroller for receiving data from the second site, and (iv) acontroller for receiving data from the third site. Each of theseseparately designated controllers may be a different controller in thecluster, or there may be overlap. For instance, different controllerscould be designated for sending state data to different sites, and forthe same remote site, different controllers could be designated forsending state data to the remote site and for receiving state data fromthe remote site.

FIG. 28 conceptually illustrates such an example showing the flow oflogical network state data between designated sender and receivermasters at three sites 2805-2815. As shown, the controller cluster ateach site includes three controllers. The first site 2805 includescontrollers 2820-2830, the second site 2810 includes controllers2835-2845, and the third site 2815 includes controllers 2850-2860.

In this example, the controllers at the first site have designated thefirst controller 2820 as the master for sending logical network statedata to the second site 2810 and the third controller 2830 as the masterfor sending logical network state data to the third site 2815. Inaddition, the second controller 2825 is designated as the master forreceiving logical network state data from the second site 2810 and thethird controller 2830 is designated as the master for receiving logicalnetwork state data from the third site 2815. Thus, at the first site,different controllers are the sender and recipient masters with respectto the second site 2810, but the same controller 2830 is both the senderand recipient master with respect to the third site 2815.

The controllers at the second site have designated the first controller2835 as the master for sending logical network state data to the firstsite 2805 and the third controller 2845 as the master for sendinglogical network state data to the third site 2815. In addition, thefirst controller 2835 is designated as the master for receiving logicalnetwork state data from the first site 2805 and the second controller2840 is designated as the master for receiving logical network statedata from the third site 2815. The same controller 2835 is the senderand recipient master with respect to the first site 2805, though theselogical network data exchanges are with two different controllers at thefirst site.

The controllers at the third site have designated the second controller2855 as the master for sending logical network state data to the firstsite 2805 and as the master for sending logical network state data tothe second site 2810. In addition, this second controller 2855 isdesignated as the master for receiving logical network state data fromthe first site 2805, while the first controller 2850 is designated asthe master for receiving logical network state data from the second site2810. Thus, the logical network data exchange between the first site2805 and the third site 2815 is bidirectional, in that the samecontrollers 2830 and 2855 are both sender and recipient masters at bothsites.

In some embodiments, the logical network state data in some embodimentsincludes logical network address to physical network address (physicallocation) mapping data as well as security group information (e.g.,network addresses of logical network endpoint DCNs belonging to securitygroups). The logical network to physical network mappings, in someembodiments, comprises mappings of logical network layer 2 (e.g., MAC)addresses to physical network tunnel endpoint layer 3 (e.g., IP)addresses at which those logical network addresses can be reached.

FIG. 29 conceptually illustrates the generation and transmission of thislogical network to physical network mapping data both within a site andbetween sites. In this example, a logical switch is stretched betweentwo sites 2905 and 2910, with central control plane (CCP) 2915 at thefirst site 2905 and CCP 2920 at the second site 2910. In this figure,the CCP at each site represents the entire controller cluster at thesite, including the distributed database and local site records. Thatis, this figure does not show any of the (potential) internal datatransactions between different nodes of the controller cluster, or thesegregation of data between information received from the computingdevices at the local site and information received from remote sites.

The first site 2905 also includes two host computers 2925 and 2930, eachof which host VMs attached to the stretched logical switch (with logicalMAC addresses A and B, respectively). The first host computer 2925 has avirtual tunnel endpoint (VTEP) interface with IP address X and thesecond host computer 2930 has a VTEP interface with IP address Y forencapsulated data messages within the first site 2905. In addition, thefirst site includes an edge device 2935 that implements a logicalnetwork gateway for the stretched logical switch. The logical networkgateway is used for processing data messages transmitted between sitesaccording to the stretched logical switch. The edge device 2935 has aVTEP interface with IP address Z for sending data messages to andreceiving data messages from host computers within the site 2905 as wellas a remote tunnel endpoint (RTEP) interface with IP address Q forsending data messages to and receiving data messages from edge devicesimplementing logical network gateways for the stretched logical switchat other sites. Logical network gateways and the use of VTEP and RTEP IPaddresses for intra-site and inter-site data traffic is described inmore detail in the concurrently filed U.S. patent application Ser. No.16/906,891, now published as U.S. Patent Publication 2021/0314192,entitled “Architecture for Stretching Logical Switches Between MultipleDatacenters”, and which is incorporated herein by reference.

The second site 2910 also includes a host computer 2940 that hosts a VM.This VM is not attached to the same stretched logical switch, but isattached to a different logical switch such that, e.g., the VM on thehost computer 2940 can communicate with the VMs attached to thestretched logical switch via a T1 logical router. As such, the span ofthe stretched logical switch includes the host computer 2940 (i.e., thehost computer receives logical network state data regarding thestretched logical switch). In addition, the second site includes an edgedevice 2945 that implements a logical network gateway for the stretchedlogical switch. This edge device 2945 has a VTEP interface with IPaddress W for sending data messages to and receiving data messages fromhost computers within the site 2910 as well as an RTEP interface with IPaddress R for sending data messages to and receiving data messages fromedge devices implementing logical network gateways for the stretchedlogical switch at other sites (e.g., the edge device 2935).

In some embodiments, when a logical network endpoint DCN (e.g., the VMson the host computers 2925 and 2930) is created on a host computer, thathost computer reports the new DCN along with data about the DCN to oneof the network controllers of the cluster. This data includes the MACaddress of the DCN, which is mapped to a virtual tunnel endpoint (VTEP)of the host computer, as well as the logical switch with which the MACaddress is associated. As shown, the host computer 2925 reports to theCCP 2915 the mapping of logical MAC address A to VTEP IP address X(i.e., that MAC address A on the stretched logical switch is located atVTEP IP address X) and the host computer 2930 reports to the CCP 2915the mapping of logical MAC address B to VTEP IP address Y (i.e., thatMAC address B on the stretched logical switch is located at VTEP IPaddress Y).

The controller cluster provides this logical MAC address to VTEP IPaddress mapping (in the context of the logical switch) to any other hostcomputers in the same site that implement the logical switch, so thatphysical forwarding elements on these other host computers can transmitlogical network data messages through the site's physical network to thelogical network DCN. Thus, the CCP 2915 at the first site 2905 providesthe mapping of MAC B to VTEP Y to the host computer 2925 and providesthe mapping of MAC A to VTEP X to the host computer 2930 (both of thesesets of data also including the logical switch context). Logical networkdata messages within a site are sent via a tunnel between the VTEP onthe host computer for the source logical network endpoint DCN and theVTEP on the host computer for the destination logical network endpointDCN, so this allows the MFEs at host computers 2925 and 2930 to transmitdata messages between the two VMs to each other through the site'snetwork. In addition, the CCP 2915 provides both of these mappings tothe edge device 2935 so that the logical network gateway can transmitany data messages for either of the two VMs that it receives from otherlogical network gateways for the stretched logical switch at otherdatacenters to the correct host computer.

The controller cluster for a particular site also provides the list oflogical MAC addresses associated with a logical switch to each othersite spanned by the logical switch. To send a data message associatedwith a logical switch from a source host computer at one site to adestination host computer at another site (e.g., from host computer 2940to host computer 2925), the source host computer (e.g., host computer2940) tunnels the data message to a first edge device implementing alogical network gateway for the logical switch in the source site (e.g.,edge device 2945), which tunnels the data message to a second edgedevice implementing a logical network gateway for the logical switch inthe destination site (e.g., edge device 2935), which in turn tunnels thedata message to the destination host computer in the second site (e.g.,host computer 2925).

As such, the CCP 2915 in the first site 2905 does not provide thelogical MAC address to VTEP mappings for the logical switch to thecontrollers in the first site, but instead provides a mapping of thelist of logical MAC addresses associated with the logical switch andlocated at the second site to RTEP IP addresses for reaching the logicalnetwork gateways at the second site. In this case, that data mapslogical MAC addresses A and B to the RTEP Q. It should be noted that insome embodiments, rather than a single RTEP IP address, this is providedas an RTEP group record with multiple RTEP IP addresses, as there are atleast two edge devices implementing logical network gateways at the sitefor the stretched logical switch (e.g., in active-standbyconfiguration).

The controllers at the first site provide this logical network statedata to the edge devices implementing the logical network gateways forthe logical switch at the first site. That is, the CCP 2920 provides theedge device 2945 with the mapping information for sending data messageswith destination MAC addresses A or B (and in the context of thestretched logical switch) to the correct edge device 2935 (e.g., via aninter-site network).

In addition, to the host computers implementing the logical switch atthe first site, the controllers provide the list of MAC addresseslocated at any of the sites (other than the first site), along with VTEPIP addresses at which edge devices implementing the logical networkgateways for the logical switch at the first site can be reached. Inthis example, the host computer 2940 receives data mapping MAC addressesA and B to VTEP W for edge device 2945 (if the logical switch wasstretched to a third site, any MAC addresses attached to the logicalswitch at that third site would be included in the same record). As withthe RTEP groups, in some embodiments these records are actually for VTEPgroups that include the different VTEP IP addresses of all logicalnetwork gateways for the stretched logical switch in the site 2910.

While this figure illustrates only data being sent from the first site2905 to the second site 2910, if any logical network endpoint DCNsattached to the stretched logical switch were hosted at the second site2910, then the logical network state data for these DCNs would be sentin the opposite direction. In addition, in the context of the logicalswitch to which the VM on host computer 2940 was attached, the CCP 2920provides this information to the CCP 2915 so that host computers andedge devices in the site 2905 can transmit data messages to the secondsite 2910 for this VM.

In addition to the logical network to physical network mapping data, thenetwork controllers of some embodiments generate and share between siteslists of logical network endpoint DCNs that belong to dynamic securitygroups. Specifically, in some embodiments the controllers receivedefinitions of dynamic security groups and use information received fromthe host computers at their site to determine the network addresses(e.g., MAC and IP addresses) for each dynamic security group that spansto the site.

FIGS. 30A-B conceptually illustrate the generation and transmission ofthese lists of logical network addresses for dynamic security groupsboth within a site and between sites. In this example, two sites 3005and 3010 are illustrated. The first site 3005 includes a local manager3015 and a CCP cluster 3020 (as in the previous figure, the internaldata transfer among the nodes of the cluster is not shown). The secondsite 3010 includes a local manager 3025 and a CCP cluster 3030. Inaddition, the first site 3005 includes two relevant host computers 3035(hosting VM A) and 3040 (hosting VM B), while the second site 3010includes one relevant host computer 3045 (hosting VM C).

From the local managers 3015 and 3025, the CCPs 3020 and 3030respectively receive definitions 3050 and 3055 of dynamic securitygroups. In some embodiments, these dynamic security group definitionsinclude security groups defined at the global manager for the globallogical network as well as any security groups defined at the respectivelocal manager. In some embodiments, security groups may be definedstatically (e.g., as a pre-specified list of MAC and/or IP addresses) ordynamically (e.g., as a set of criteria). The controllers areresponsible for determining which local logical network endpoint DCNsbelong to each dynamic security group. The criteria for belonging to adynamic security group may vary in different embodiments. For instance,these criteria can include attachment to a specific logical switch, anIP address in a particular subnet, the operating system running on aparticular VM, the site at which a DCN is located, the type ofapplication operating on a DCN, etc. In this example, the securitygroups include two groups (Group 1 and Group 2) that span both sites3005 and 3010, as well as a third group (Group 3) that only spans thesecond site 3010.

As mentioned with respect to FIG. 29 , in some embodiments when alogical network endpoint DCN (e.g., any of VMs A-C) is created on a hostcomputer, that host computer reports the new DCN along with data aboutthe DCN to one of the network controllers of the cluster. This dataincludes not only the MAC and IP addresses of the VM, but informationabout the logical switch to which the DCN attaches as well as variousother runtime state data for the DCN in some embodiments. As shown, hostcomputer 3035 reports the attachment of VM A and host computer 3040reports the attachment of VM B to the CCP 3020, while host computer 3045reports the attachment of VM C to the CCP 3030.

When a logical network endpoint DCN matches the set of criteria for aparticular security group, the controller adds the logical networkaddresses (e.g., MAC and IP addresses) for the DCN to the securitygroup. In some embodiments, the controllers use information receivedfrom a host computer when the DCN is created on the host computer to (i)identify to which groups the DCN belongs and (ii) identify the MAC andIP addresses to add to the lists for the identified groups. The CCP 3020identifies that both VM A and VM B matches the criteria for Group 1,while only VM A matches the criteria for Group 2. At the second site3010, the CCP 3030 determines that VM C belongs to Group 1 and Group 3(but not to Group 2). These CCPs store this information in theirrespective storages 3060 and 3065. In some embodiments, these storages3060 and 3065 represent the amalgamation of the stored security groupinformation at each of the controller clusters. As described above, insome embodiments the data storing lists of network addresses for eachsecurity group is stored separately for each site (i.e., differentstorages for each remote site as well as for the local site).

For each group spanning multiple sites, the controller clusters at thosesites share the list of logical network addresses belonging to the groupwith each other. The controllers then provide the full list of addressesfor each group to the host computers and/or edge devices that enforcepolicy rules using the security groups. As shown in FIG. 30A, the CCP3020 at the first site 3005 provides data to the CCP 3030 at the secondsite 3010 indicating the network addresses located in the first site3005 for both of Group 1 and Group 2. Correspondingly, the CCP 3030 atthe second site 3010 provides data to the CCP 3020 at the first site3005 indicating the network addresses located in the second site forGroup 1. No data is exchanged for Group 2 because the VM in the secondsite 3010 does not belong to this security group, and no data isexchanged for Group 3 because this security group does not span to thefirst site 3005.

FIG. 30B illustrates that the CCPs 3020 and 3030 have updated theirrespective storages 3060 and 3065 to include the logical network statedata received from the other site (e.g., by adding this information totheir respective distributed databases for storing data from therespective remote site). In addition, this information is provided toeach of the host computers at their respective sites. The host computers3035 and 3040 at the first site 3005 receive information about thecomplete group membership for Groups 1 and 2, while the host computer3045 at the second site 3010 receives information about the completegroup membership for Groups 1, 2, and 3.

In the previous examples, the controller clusters send all of the statefrom their respective sites at once. While this is plausible for thesimple examples shown in these figures, realistic examples may havehundreds or thousands of network addresses associated with a singlesecurity group or logical switch in one site, with many differentsecurity groups and/or logical switches for which network state dataneeds to be synchronized between sites. In such a situation, updateswill occur frequently, and it would be very bandwidth-intensive totransfer the entire logical network state with each update.

Instead, when providing updates to the logical network state data, someembodiments send each change to the current state as an atomic updatespecifying the change, thereby minimizing the amount of data that needsto be transferred between sites. The controllers at a particular sitemaintain a snapshot of the current logical network state (e.g., in thedistributed database at the site), and whenever this state changes(e.g., due to creation or deletion of a DCN from a host computer in thesite), each controller that handles sending that state to another siteidentifies the change and sends the change as an update to the othersite. Because these changes can be derived by any of the controllers inthe cluster at the sending site, this site does not persist the queue ofupdates in some embodiments. In some embodiments, the synchronizationprotocol is lossless, so once an update is sent from a first site to asecond site it can be assumed that the second site will process thatupdate (and will do so in the order that the updates are received). Inaddition, the controllers at the second site persist these updates in adistributed database.

FIGS. 31A-B conceptually illustrate such an update to a dynamic securitygroup at a first site and the persisting of updates at a second siteover four stages 3105-3117. The first stage 3105 illustrates the CCPs3120 and 3125 at two different sites, and their respective storages 3130and 3135 for the logical network state data indicating the membership ofa security group (Group 1) at the first site. In some embodiments, eachcontroller cluster separately stores the logical network state data forits own site and for each site from which the state data is received.For instance, some embodiments store the list of security group members(e.g., IP and MAC addresses) in their own site persisted in a database(or persist the data received from local computing devices, from whichthe list of security group members can be generated), and also persist aqueue of updates for each other site. The queue of updates, in differentembodiments, may be separate queues for updates from each other site, ora single queue for all of the sites with tags that indicate from whichsite each update was received. Thus, the database 3130 stores a list ofcurrent members of Group 1 (A and B) at the first site, while thedatabase 3135 stores a set of updates to Group 1. These updates specifyto Add D to the security group, Add A to the security group, Add B tothe security group, and finally to delete D from the security group.

In the first stage 3105, a host computer 3140 reports to the CCP 3120the creation of a new logical network endpoint DCN C on the hostcomputer 3140. As a result, the CCP 3120 determines to which dynamicsecurity groups the DCN C belongs, which in this case includes Group 1.As shown in the second stage 3110, the CCP 3120 adds the networkaddresses for DCN C to Group 1 in its database 3130. Though not shown,the CCP 3120 would provide this information to the host computer 3140.In addition, as shown in the second stage 3110, the CCP 3120 at thefirst site sends an update 3145 to the CCP 3125 at the second site,specifying to add the network addresses for DCN C to Group 1.

In response to receiving the update 3145, in the third stage 3115 theCCP 3125 adds the update adding the network addresses for DCN C to Group1 in its persisted queue of updates for Group 1 in the database 3135.This data, stored as a series of updates, now is equivalent to the datastored in the database 3130. Though not shown, the CCP 3125 wouldsubsequently provide this update to any host computers at the secondsite that require the membership of Group 1 for enforcing policies.

The third stage 3115 also illustrates that, after a period of time, theCCP 3125 removes the updates to Add D and delete D from the update queue3135. In some embodiments, the CCP removes logical network state dataupdates that are no longer relevant (i.e., when an update specifies todelete an entity from a list, all previous updates regarding that entitycan be removed. Different embodiments may perform these operationsperiodically, upon receiving the delete update, etc. The fourth stage3117 illustrates that the update queue 3135 now only includes theupdates for DCNs A, B, and C. It should be noted that while this exampleshows an update to security group membership, some embodiments treatupdates to other logical network state data (e.g., logical network tophysical network mapping records) in the same manner.

If the connection from a first (sending) site to a second (receivingsite) goes down, some of the updates generated at the first site may notbe sent to the second site. The CCP at the first site may not know whichupdates need to be sent to the second site (e.g., because these updatesare not persisted at the first site, and there is no acknowledgmentprocess for each individual update). However, sending the entire logicalnetwork state to be synchronized with the second site is an inefficientuse of bandwidth, as mentioned above. Instead, some embodiments compare(i) a snapshot of the persisted data at the second site and (ii) asnapshot of the state at the first site to identify the differences. Thefirst site can thus only send these differences to the second site.

FIG. 32 conceptually illustrates a process 3200 of some embodiments foridentifying logical network state data updates required to be sent to acontroller at a remote site after reconnection with the remote site. Insome embodiments, the process 3200 is performed by a network controllercluster at a first site (e.g., the network controller responsible forsending logical network state data to a second, remote site).

As shown, the process 3200 begins by detecting (at 3205) reconnectionwith a receiver controller from another site after a period ofdisconnect. In some embodiments, the controllers use a heartbeatprotocol or other mechanism to detect when the connection withcontrollers at other sites is down or has come back up. This connectioncould be down due to a physical connection issue between the sites, anissue with the controller nodes at the remote site, or other reasons.

After reconnection, the process 3200 receives (at 3210) a cryptographichash tree from the receiver controller based on a snapshot of thelogical network state generated from the persisted data at the othersite. In some embodiments, as discussed above, this persisted data is aqueue of updates, which can be used to generate the logical networkstate (e.g., a list of MAC addresses at the sender site associated witha particular logical switch and the RTEP IP addresses for the sendersite, a list of network addresses belonging to a particular securitygroup at the sender site). In some embodiments, the recipient controlleruses this logical network state to generate a cryptographic hash tree(e.g., a Merkle tree) and sends this hash tree to the sendingcontroller. In some embodiments, the hash tree leaf nodes are hashes ofdata blocks (e.g., logical network state data), and subsequent parentnodes are hashes of their children.

The process 3200 also computes (at 3215) a cryptographic hash tree basedon a snapshot of the logical network state stored in the local sitedatabase (i.e., for the logical network state data that should berepresented at the second site). This data is already stored in thedistributed database for the controller cluster, and the same algorithmis used to compute the hash tree as was performed at the receiver.

Next, the process 3200 compares (at 3220) the two hash trees to identifythe differences in the logical network state. In some embodiments, ifthe root node is the same, this is (effectively) a guaranteed that theentire state is the same, and no updates are required. If there aredifferences in the root node, the process can continue towards the leafnodes to identify the differences. If any node is the same between thetwo hash trees, then any child nodes will be the same. Using thisprocess, the initial data blocks that are different can be identified,and thus the differences in logical network state between the two sitesare identified. Based on this comparison, the process 3200 sends (at3225) any required updates to the remote site.

Another consequence of a connection problem can be a conflict betweenlogical network state data from two different sites. Specifically, whena logical network endpoint DCN moves from a first site to a second site,conflicts may arise at a third site based on data received from thefirst two sites. When a DCN is migrated from the first site to thesecond site, or crashes in the first site and is brought back up in thesecond site, in an ideal scenario (i) the controller at the first siteis notified of the deletion of the DCN, updates its logical networkstate data accordingly, and shares this data with the third site (andthe second site), and (ii) the controller at the second site is notifiedof the DCN creation, updates its logical network state data accordingly,and shares this data with the third site (and the first site). However,if there is a connection problem at the second site (e.g., between thehost computer and the controller cluster or between the first and secondsites), then the first site will not receive information about thedeletion, and will thus end up with conflicting information (e.g., IPaddresses in a security group) once the information from the third siteis received.

FIG. 33 conceptually illustrates a process 3300 of some embodiments forresolving such conflicting logical network state data. The process 3300is performed by a controller at one site upon detecting conflictinglogical network state data between data received from two other sites.

As shown, the process 3300 begins by detecting (at 3305) a conflictbetween logical network state data from two different sites. An exampleof such a conflict could be if a controller from a first site sends anupdate that a particular IP address should be added to a security groupand is located at the first site, when that particular IP address hasalready been added to the security group based on logical network statedata from a second state (and has not been deleted from the securitygroup). In some cases, the MAC address for a DCN will change when theDCN is moved between sites, but the IP address will stay the same. Ifthe MAC address is also the same, then a conflict in the logical networkto physical network mapping records may also occur.

Upon detecting the conflict, the process 3300 determines (at 3310)whether the connection to controllers at more than one site that is thesource of conflicting data is available. While the examples abovedescribe conflicts between information from two sites, it is possible inrare situations to have conflicts between more than two sites.

If the connection to the controllers at only one site is available, thenthe process 3300 uses (at 3315) the logical network state data from thesite with the available connection on the presumption that this logicalnetwork state data is up to date and the controllers at the othersite(s) would provide an update resolving the conflict if the connectionwas available. On the other hand, if the connection is available withmultiple sites, then the process 3300 uses (at 3320) the most recentlogical network state data based on timestamps appended to the updates.It may also be the case that a DCN was removed from a host in one of thesites but, due to a loss of connection between the host and thecontrollers at that site, the controller cluster was not made aware ofthe removal and thus could not generate or send an update. In this case,the creation of the new DCN at a different site will be more recent andthus should be used.

FIG. 34 conceptually illustrates an electronic system 3400 with whichsome embodiments of the invention are implemented. The electronic system3400 may be a computer (e.g., a desktop computer, personal computer,tablet computer, server computer, mainframe, a blade computer etc.),phone, PDA, or any other sort of electronic device. Such an electronicsystem includes various types of computer readable media and interfacesfor various other types of computer readable media. Electronic system3400 includes a bus 3405, processing unit(s) 3410, a system memory 3425,a read-only memory 3430, a permanent storage device 3435, input devices3440, and output devices 3445.

The bus 3405 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 3400. For instance, the bus 3405 communicativelyconnects the processing unit(s) 3410 with the read-only memory 3430, thesystem memory 3425, and the permanent storage device 3435.

From these various memory units, the processing unit(s) 3410 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 3430 stores static data and instructions thatare needed by the processing unit(s) 3410 and other modules of theelectronic system. The permanent storage device 3435, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system3400 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 3435.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 3435, the system memory 3425 is a read-and-write memorydevice. However, unlike storage device 3435, the system memory is avolatile read-and-write memory, such a random-access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 3425, the permanent storage device 3435, and/or theread-only memory 3430. From these various memory units, the processingunit(s) 3410 retrieve instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 3405 also connects to the input and output devices 3440 and3445. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 3440 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 3445 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 34 , bus 3405 also couples electronic system3400 to a network 3465 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 3400 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra-density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral signals.

This specification refers throughout to computational and networkenvironments that include virtual machines (VMs). However, virtualmachines are merely one example of data compute nodes (DCNs) or datacompute end nodes, also referred to as addressable nodes. DCNs mayinclude non-virtualized physical hosts, virtual machines, containersthat run on top of a host operating system without the need for ahypervisor or separate operating system, and hypervisor kernel networkinterface modules.

VMs, in some embodiments, operate with their own guest operating systemson a host using resources of the host virtualized by virtualizationsoftware (e.g., a hypervisor, virtual machine monitor, etc.). The tenant(i.e., the owner of the VM) can choose which applications to operate ontop of the guest operating system. Some containers, on the other hand,are constructs that run on top of a host operating system without theneed for a hypervisor or separate guest operating system. In someembodiments, the host operating system uses name spaces to isolate thecontainers from each other and therefore provides operating-system levelsegregation of the different groups of applications that operate withindifferent containers. This segregation is akin to the VM segregationthat is offered in hypervisor-virtualized environments that virtualizesystem hardware, and thus can be viewed as a form of virtualization thatisolates different groups of applications that operate in differentcontainers. Such containers are more lightweight than VMs.

Hypervisor kernel network interface modules, in some embodiments, is anon-VM DCN that includes a network stack with a hypervisor kernelnetwork interface and receive/transmit threads. One example of ahypervisor kernel network interface module is the vmknic module that ispart of the ESXi™ hypervisor of VMware, Inc.

It should be understood that while the specification refers to VMs, theexamples given could be any type of DCNs, including physical hosts, VMs,non-VM containers, and hypervisor kernel network interface modules. Infact, the example networks could include combinations of different typesof DCNs in some embodiments.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 13, 14, 17, 27, 32, and 33 ) conceptually illustrateprocesses. The specific operations of these processes may not beperformed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process. Thus, one ofordinary skill in the art would understand that the invention is not tobe limited by the foregoing illustrative details, but rather is to bedefined by the appended claims.

We claim:
 1. A network management system for managing a logical networkthat spans a plurality of physical sites, the network management systemcomprising: a global network manager for receiving global logicalnetwork configuration data for the plurality of physical sites; and ateach of the physical sites: a set of computing devices, each computingdevice executing (i) a set of forwarding elements to implement thelogical network at the physical site and (ii) a local controller forconfiguring the set of forwarding elements on the computing device; alocal network manager for receiving a logical network configurationspecific to the physical site from the global network manager; and a setof central controllers for distributing logical network configurationdata based on the logical network configuration specific to the physicalsite received by the global manager to the local controllers thatexecute on the computing devices at the physical site.
 2. The networkmanagement system of claim 1, wherein the global network managerexecutes at a particular one of the physical sites of the plurality ofphysical sites.
 3. The network management system of claim 1, wherein theglobal logical network configuration comprises a set of policiesdefining elements of the logical network.
 4. The network managementsystem of claim 3, wherein the logical network configuration specific toa particular physical site comprises a subset of the policies thatpertain to the particular physical site.
 5. The network managementsystem of claim 4, wherein the logical network configuration datadistributed to the computing devices at a particular physical site isgenerated based on the subset of the policies that pertain to theparticular physical site.
 6. The network management system of claim 1,wherein the set of central controllers at a particular physical sitereceives logical network to physical network mapping data from the setof computing devices that implement the logical network at theparticular physical site.
 7. The network management system of claim 6,wherein the set of central controllers at the particular physical site(i) provides the logical network to physical network mapping data to thesets of central controllers at the other physical sites and (ii)receives logical network to physical network mapping data from the setsof central controllers at the other physical sites.
 8. The networkmanagement system of claim 1, wherein the global network managerreceives the global logical network configuration data from a networkadministrator.
 9. The network management system of claim 8, wherein thelocal network manager for a particular physical site (i) receives thelogical network configuration specific to the physical site from theglobal network manager based on the global logical network configurationand (ii) receives additional local logical network configuration for theparticular physical site from the network administrator.
 10. The networkmanagement system of claim 9, wherein the network administrator uses asame client to access the global network manager and the local networkmanager.
 11. A network management system for managing a logical networkthat spans a plurality of physical sites, the network management systemcomprising: a global network manager for receiving global logicalnetwork configuration data for the plurality of physical sites; and ateach of the physical sites: a local network manager for receiving alogical network configuration for the physical site from the globalnetwork manager; and a set of central controllers for distributinglogical network configuration data to computing devices that implementthe logical network at the physical site, wherein the global networkmanager executes at a separate site than any of the physical sitesspanned by the logical network.
 12. The network management system ofclaim 11, wherein: the computing devices at each physical site eachexecute (i) a set of forwarding elements to implement the logicalnetwork at the physical site and (ii) a local controller for configuringthe set of forwarding elements on the computing device, and the set ofcentral controllers at each physical site distributes logical networkconfiguration data to the local controllers that execute on thecomputing devices at the physical site based on the logical networkconfiguration specific to the physical site received by the localmanager at the physical site.
 13. A network management system formanaging a logical network that spans a plurality of physical sites, thenetwork management system comprising: a global network manager,executing at a particular one of the physical sites, for receivingglobal logical network configuration data for the plurality of physicalsites; and at each of the physical sites: a local network manager forreceiving a logical network configuration for the physical site from theglobal network manager, wherein the global network manager executes on asame computing device as the local network manager for the particularphysical site; and a set of central controllers for distributing logicalnetwork configuration data to computing devices that implement thelogical network at the physical site.
 14. The network management systemof claim 13, wherein the global network manager and the local networkmanager for the particular physical site share a database stored at theparticular physical site.
 15. A network management system for managing alogical network that spans a plurality of physical sites, the networkmanagement system comprising: an active global network manager forreceiving global logical network configuration data for the plurality ofphysical sites; a standby global network manager; and at each of thephysical sites: a local network manager for receiving a logical networkconfiguration for the physical site from the global network manager; anda set of central controllers for distributing logical networkconfiguration data to computing devices that implement the logicalnetwork at the physical site.
 16. The network management system of claim15, wherein the active global manager executes at a first site of theplurality of physical sites and the standby global manager executes at asecond site of the plurality of physical sites.
 17. The networkmanagement system of claim 15, wherein each of the active and standbyglobal network managers executes on a cluster of machines.
 18. A networkmanagement system for managing a logical network that spans a pluralityof physical sites, the network management system comprising: a globalnetwork manager for receiving global logical network configuration datafor the plurality of physical sites; and at each of the physical sites:a local network manager for receiving a logical network configurationfor the physical site from the global network manager, wherein thelogical network configuration for a particular physical site comprisesat least one security group definition specifying a set of criteria forbelonging to the security group; and a set of central controllers fordistributing logical network configuration data to computing devicesthat implement the logical network at the physical site, wherein the setof central controllers at the particular physical site translates thesecurity group definition into a set of network addresses of logicalnetwork endpoints that match the set of criteria at the particularphysical site.
 19. The network management system of claim 18, whereinthe set of central controllers at the particular physical site (i)provides the set of network addresses to the sets of central controllersat at least a subset of the other physical sites and (ii) receives setsof network addresses of logical network endpoints that match the set ofcriteria at the other physical sites from the sets of centralcontrollers at the other physical sites.
 20. A network management systemfor managing a logical network that spans a plurality of physical sites,the network management system comprising: a global network manager forreceiving global logical network configuration data for the plurality ofphysical sites, the global logical network configuration comprising aset of policies defining elements of the logical network; and at each ofthe physical sites: a local network manager for receiving a logicalnetwork configuration for the physical site from the global networkmanager, wherein the logical network configuration for a particularphysical site comprises a subset of the policies that pertain to theparticular physical site; and a set of central controllers fordistributing logical network configuration data to computing devicesthat implement the logical network at the physical site, wherein, at theparticular physical site: the local network manager converts the subsetof the policies into the logical network configuration data and providesthe logical network configuration data to the set of centralcontrollers; and the set of central controllers determine whichcomputing devices require which logical network configuration data anddistribute the appropriate logical network configuration data to thecomputing devices.
 21. A network management system for managing alogical network that spans a plurality of physical sites, the networkmanagement system comprising: a global network manager for receivingglobal logical network configuration data for the plurality of physicalsites; and at each of the physical sites: a local network manager forreceiving a logical network configuration for the physical site from theglobal network manager; and a set of central controllers fordistributing logical network configuration data to computing devicesthat implement the logical network at the physical site, wherein the setof central controllers at a first site of the plurality of physicalsites communicate with the set of central controllers at a second siteof the plurality of physical sites.
 22. A network management system formanaging a logical network that spans a plurality of physical sites, thenetwork management system comprising: a global network manager forreceiving global logical network configuration data for the plurality ofphysical sites; and at each of the physical sites: a local networkmanager for receiving a logical network configuration for the physicalsite from the global network manager; and a set of central controllersfor distributing logical network configuration data to computing devicesthat implement the logical network at the physical site, wherein thesets of central controllers at the plurality of physical sitescommunicate in a full mesh such that each respective set of centralcontrollers at a respective physical sites communicates with the sets ofcentral controllers at each of the other physical sites.